Instruction: FashionMNIST Dataset not transforming to Tensor

Input:
Trying to calculate the mean and standard deviation of the dataset to normalise it afterwards.
Current Code:
train_dataset = datasets.FashionMNIST('data', train=True, download = True, transform=[transforms.ToTensor()])
test_dataset = datasets.FashionMNIST('data', train=False, download = True, transform=[transforms.ToTensor()])

def calc_torch_mean_std(tens):
    mean = torch.mean(tens, dim=1)
    std = torch.sqrt(torch.mean((tens - mean[:, None]) ** 2, dim=1))
    return(std, mean)

train_mean, train_std = calc_torch_mean_std(train_dataset)

test_mean, test_std = calc_torch_mean_std(test_dataset)

However, i'm getting the error:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/var/folders/16/crymx03s6pzfspm_3qfrlkx00000gn/T/ipykernel_72423/605045038.py in <module>
      8     return(std, mean)
      9
---> 10 train_mean, train_std = calc_torch_mean_std(train_dataset)
     11
     12 test_mean, test_std = calc_torch_mean_std(test_dataset)

/var/folders/16/crymx03s6pzfspm_3qfrlkx00000gn/T/ipykernel_72423/605045038.py in calc_torch_mean_std(tens)
      4
      5 def calc_torch_mean_std(tens):
----> 6     mean = torch.mean(tens, dim=1)
      7     std = torch.sqrt(torch.mean((tens - mean[:, None]) ** 2, dim=1))
      8     return(std, mean)

TypeError: mean() received an invalid combination of arguments - got (FashionMNIST, dim=int), but expected one of:
 * (Tensor input, *, torch.dtype dtype)
 * (Tensor input, tuple of ints dim, bool keepdim, *, torch.dtype dtype, Tensor out)
 * (Tensor input, tuple of names dim, bool keepdim, *, torch.dtype dtype, Tensor out)

It should be getting a tensor as i transform the data as it comes in using transforms.ToTensor().
Checked import of transforms and it is okay. Checked parameters for the datasets.FashionMNIST() and transform is correctly used (should work both with and without [ ]).
Expecting no error, and to get the mean and std for both datasets.

Output:
datasets.FashionMNIST returns (image, target) where target is index of the target class. So if you want to take the mean you need to extract just the image.
images = torch.vstack([pair[0] for pair in train_dataset])

images should now be of shape (N, H, W) and you can do whatever you want from there.
Another solution as noted by OP is to use train_dataset.data to directly access the data.


Instruction: conv2d() received an invalid combination of arguments

Input:
After resnet convolution, I want to further compress the 256 dimensions to 20 dimensions. I directly wrote a layer in the back, but after forward propagation, there is an error in this layer, I don't know why？
def forward(self, x):
    x = self.conv1(x)
    dif_residual1 = self.downsample1(x)
    x = self.layer1_1(x)
    x =x + dif_residual1
    residual = x
    x = self.layer1_2(x)
    x = x + residual
    residual = x
    x = self.layer1_3(x)
    x = x + residual

    if self.out_channel != 256:
        x = self.layer2

    filters = torch.ones(self.batch_size, self.out_channel, 1, 1).detach().requires_grad_(False).to(self.device)
    x = F.conv2d(x, weight=filters, padding=0)

The dimension of x before I do if is：
x = {Tensor:(1,256,117,240)}

But after the if statement is executed, it becomes what the picture shows。
The error I get is this：
x = F.conv2d(feature, weight=filters, padding=0)
TypeError: conv2d() received an invalid combination of arguments - got (Sequential, weight=Tensor, padding=int), but expected one of:
 * (Tensor input, Tensor weight, Tensor bias, tuple of ints stride, tuple of ints padding, tuple of ints dilation, int groups)
 * (Tensor input, Tensor weight, Tensor bias, tuple of ints stride, str padding, tuple of ints dilation, int groups)

Encounter a new problem：
File "D:\software\Anaconda\envs\torch1.10\lib\site-packages\torch\autograd\__init__.py", line 173, in backward
Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1, 1, 117, 240]], which is output 0 of AddBackward0, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
My code：
class VGG(nn.Module):

def __init__(self, in_channel, out_channel=None, init_weights=True, device='gpu',batch_size=1):
    super(VGG, self).__init__()

    self.batch_size = batch_size
    self.out_channel = out_channel
    if device == 'gpu':
        self.device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
    else:
        self.device = torch.device("cpu")

    modes = 'reflect'
    out_channel1 = 64
    self.conv1_1 = nn.Sequential(
        nn.Conv2d(in_channels=in_channel, out_channels=out_channel1, kernel_size=3, stride=1, padding=1, padding_mode = modes, bias=False),
        nn.BatchNorm2d(out_channel1),
        nn.LeakyReLU()
    )
    self.conv1_2 = nn.Sequential(
        nn.Conv2d(in_channels=out_channel1, out_channels=out_channel1, kernel_size=3, stride=1, padding=1, padding_mode = modes, bias=False),
        nn.BatchNorm2d(out_channel1),
        nn.LeakyReLU()
    )
    out_channel2 = 128
    self.conv2_1 = nn.Sequential(
        nn.Conv2d(in_channels=out_channel1, out_channels=out_channel2, kernel_size=3, stride=1, padding=1, padding_mode = modes, bias=False),
        nn.BatchNorm2d(out_channel2),
        nn.LeakyReLU()
    )
    self.conv2_2 = nn.Sequential(
        nn.Conv2d(in_channels=out_channel2, out_channels=out_channel2, kernel_size=3, stride=1, padding=1, padding_mode = modes, bias=False),
        nn.BatchNorm2d(out_channel2),
        nn.LeakyReLU()
    )
    out_channel3 = 256
    self.conv3_1 = nn.Sequential(
        nn.Conv2d(in_channels=out_channel2, out_channels=out_channel3, kernel_size=3, stride=1, padding=1, padding_mode = modes, bias=False),
        nn.BatchNorm2d(out_channel3),
        nn.LeakyReLU()
    )
    self.conv3_2 = nn.Sequential(
        nn.Conv2d(in_channels=out_channel3, out_channels=out_channel3, kernel_size=3, stride=1, padding=1, padding_mode = modes, bias=False),
        nn.BatchNorm2d(out_channel3),
        nn.LeakyReLU()
    )

    if out_channel == None:
        self.out_channel = 256
        self.conv3_3 = nn.Sequential(
            nn.Conv2d(in_channels=out_channel3, out_channels=out_channel3, kernel_size=3, stride=1, padding=1,
                      padding_mode=modes, bias=False),
            nn.BatchNorm2d(out_channel3),
            nn.LeakyReLU()
        )
    else:
        self.conv3_3 = nn.Sequential(
            nn.Conv2d(in_channels=out_channel3, out_channels=out_channel3, kernel_size=3, stride=1, padding=1, padding_mode=modes, bias=False),
            nn.BatchNorm2d(out_channel3),
            nn.LeakyReLU(),
            nn.Conv2d(in_channels=out_channel3, out_channels=out_channel, kernel_size=3, stride=1, padding=1, padding_mode=modes, bias=False),
            nn.BatchNorm2d(out_channel),
            nn.LeakyReLU()
        )

    if init_weights:
        self._init_weight()

def forward(self, x):

    x = self.conv1_1(x)
    x = self.conv1_2(x)

    x = self.conv2_1(x)
    x = self.conv2_2(x)

    x = self.conv3_1(x)
    x = self.conv3_2(x)

    x = self.conv3_3(x)

    feature = x

    filters = torch.ones(self.batch_size, self.out_channel, 1, 1).detach().requires_grad_(False).to(self.device)
    x = F.conv2d(x, weight = filters, padding = 0)

    return x,feature

out_channel = 20
model = VGG(in_channel=12, out_channel=out_channel, init_weights=True, batch_size=batch_size)

 for epoch in range(start_epoch+1,epochs):
    # train
    model.train()
    running_loss = 0.0
    train_bar = tqdm(train_loader, file=sys.stdout)
    for step, data in enumerate(train_bar):
        images, labels = data
        optimizer.zero_grad()
        outputs,feature = model(images.to(device))
        outputs = tonser_nolmal(outputs)
        loss = loss_function(outputs, labels.to(device))
        loss.backward()
        optimizer.step()
        running_loss += loss.item()

        train_bar.desc = "train epoch[{}/{}] loss:{:.6f}".format(epoch + 1,
                                                                 epochs,
                                                                 loss)
    checkpoint = {
        "net": model.state_dict(),
        "optimizer": optimizer.state_dict(),
        "epoch": epoch
    }
    torch.save(checkpoint, save_path + "/model-{}.pth".format(epoch))

    # validate
    model.eval()
    count_acc = 0.0
    count_mae = 0.0
    with torch.no_grad():
        val_bar = tqdm(validate_loader, file=sys.stdout)
        for val_data in val_bar:
            val_images, val_labels = val_data
            outputs,_ = model(val_images.to(device))
            # outputs = F.normalize(outputs,dim=3)
            outputs = tonser_nolmal(outputs)
            loss = loss_function(outputs, val_labels.to(device))
            count_acc = count_acc + loss.item()
            mae = Evaluation().MAE(outputs, val_labels.to(device))
            count_mae = count_mae + mae.item()

Output:
The error is likely to be caused by the following variable assignment:
    if self.out_channel != 256:
        x = self.layer2

which can be easily fixed by changing it to
        x = self.layer2(x)

Update:
As OP updated his code, I did some test. There were several things which I found problematic:

self._init_weight was not provided, so I commented it out;
filters = torch.ones(self.batch_size, self.out_channel, 1, 1).detach().requires_grad_(False).to(self.device). The filter weight should have a shape of (c_out, c_in, kernel_size, kernel_size). However, batch_size appeared in the position of out_channels.
The role of filter in the forward was not clear to me. If you wanted to reduce the out_channels further from 256 to 20, then initializing your model with VGG(..., out_channel=20) is sufficient. Basically, self.conv3_3 would do the job.

On my end, I modified the code a little bit and it ran successfully:
import sys
import torch
import torch.nn as nn
from tqdm import tqdm
from torchvision.datasets import FakeData
from torch.utils.data import DataLoader
import torch.nn.functional as F

dataset = [torch.randn(12, 64, 64) for _ in range(1000)]
train_loader = DataLoader(dataset, batch_size=1, shuffle=True)

class VGG(nn.Module):

    def __init__(self, in_channel, out_channel=None, init_weights=True, device='cpu', batch_size=1):
        super(VGG, self).__init__()

        self.batch_size = batch_size
        self.out_channel = out_channel
        self.device = device

        modes = 'reflect'
        out_channel1 = 64
        self.conv1_1 = nn.Sequential(
            nn.Conv2d(in_channels=in_channel, out_channels=out_channel1, kernel_size=3, stride=1, padding=1, padding_mode = modes, bias=False),
            nn.BatchNorm2d(out_channel1),
            nn.LeakyReLU()
        )
        self.conv1_2 = nn.Sequential(
            nn.Conv2d(in_channels=out_channel1, out_channels=out_channel1, kernel_size=3, stride=1, padding=1, padding_mode = modes, bias=False),
            nn.BatchNorm2d(out_channel1),
            nn.LeakyReLU()
        )
        out_channel2 = 128
        self.conv2_1 = nn.Sequential(
            nn.Conv2d(in_channels=out_channel1, out_channels=out_channel2, kernel_size=3, stride=1, padding=1, padding_mode = modes, bias=False),
            nn.BatchNorm2d(out_channel2),
            nn.LeakyReLU()
        )
        self.conv2_2 = nn.Sequential(
            nn.Conv2d(in_channels=out_channel2, out_channels=out_channel2, kernel_size=3, stride=1, padding=1, padding_mode = modes, bias=False),
            nn.BatchNorm2d(out_channel2),
            nn.LeakyReLU()
        )
        self.out_channel3 = out_channel3 = 256
        self.conv3_1 = nn.Sequential(
            nn.Conv2d(in_channels=out_channel2, out_channels=out_channel3, kernel_size=3, stride=1, padding=1, padding_mode = modes, bias=False),
            nn.BatchNorm2d(out_channel3),
            nn.LeakyReLU()
        )
        self.conv3_2 = nn.Sequential(
            nn.Conv2d(in_channels=out_channel3, out_channels=out_channel3, kernel_size=3, stride=1, padding=1, padding_mode = modes, bias=False),
            nn.BatchNorm2d(out_channel3),
            nn.LeakyReLU()
        )

        self.out_channel = out_channel
        if out_channel == None:
            self.conv3_3 = nn.Sequential(
                nn.Conv2d(in_channels=out_channel3, out_channels=out_channel3, kernel_size=3, stride=1, padding=1,
                          padding_mode=modes, bias=False),
                nn.BatchNorm2d(out_channel3),
                nn.LeakyReLU()
            )
        else:
            self.conv3_3 = nn.Sequential(
                nn.Conv2d(in_channels=out_channel3, out_channels=out_channel3, kernel_size=3, stride=1, padding=1, padding_mode=modes, bias=False),
                nn.BatchNorm2d(out_channel3),
                nn.LeakyReLU(),
                nn.Conv2d(in_channels=out_channel3, out_channels=out_channel, kernel_size=3, stride=1, padding=1, padding_mode=modes, bias=False),
                nn.BatchNorm2d(out_channel),
                nn.LeakyReLU()
            )
        # The implementation of _init_weight is not found
#         if init_weights:
#             self._init_weight()

    def forward(self, x):

        x = self.conv1_1(x)
        x = self.conv1_2(x)

        x = self.conv2_1(x)
        x = self.conv2_2(x)

        x = self.conv3_1(x)
        x = self.conv3_2(x)

        x = self.conv3_3(x)

        feature = x

        if x.shape[1] == 256:  # self.out_channel is None
            filters = torch.ones(20, self.out_channel3, 1, 1).to(self.device)
            x = F.conv2d(x, weight = filters, padding = 0)

        return x, feature

out_channel = 20
device = "cuda:0" if torch.cuda.is_available() else "cpu"
model = VGG(in_channel=12, out_channel=None, init_weights=True, device=device, batch_size=1)
model.to(device)
print(model(next(iter(train_loader)).to(device))[0].shape)
model = VGG(in_channel=12, out_channel=20, init_weights=True, device=device, batch_size=1)
model.to(device)
print(model(next(iter(train_loader)).to(device))[0].shape)

Outputs:
torch.Size([1, 20, 64, 64])
torch.Size([1, 20, 64, 64])


Instruction: Load and Retrain PyTorch model…

Input:
Hi all!  Can you help me?  I have a question: "How can I retrain a PyTorch model with just a .pt file ?”
I've looked at many guides but haven't found the answer.  Everything there has a model class.
I tried to import using torch.load() and others.  But it doesn't work.  When I load the model with torch.load() I can't get the parameters.enter image description here
MyModel class I write myself? Doesn't it have to match the class in .pt?

Output:
To retrain a PyTorch model with just a .pt file, you will need to use the PyTorch API to create a new model instance and load the .pt file as the initial weights for the model. This can be done using the torch.load() function to load the .pt file, and then using the model.load_state_dict() method to load the weights into the model.
Here is an example of how this might be done:
import torch

# Create a new model instance
model = MyModel()

# Load the .pt file as the initial weights for the model
weights = torch.load('model.pt')
model.load_state_dict(weights)

# Retrain the model using the loaded weights as the starting point
model.fit(...)

In this example, MyModel is the class for your model, and model.pt is the .pt file containing the initial weights for the model. The model.fit() method is used to retrain the model using the loaded weights as the starting point.
It is important to note that this approach assumes that the .pt file was created using the same model class (MyModel in this example) and that the model architecture has not changed since the .pt file was created. If the model architecture has changed, you will need to make sure that the new model architecture is compatible with the weights in the .pt file, or you may need to use a different approach to retrain the model.


Instruction: In pytorch torchscript, how to define mutiple entry point

Input:
Have a torch model as follow :
  MyModel
       update(self)    :  Update some params.
       predict(self,X) :  Predict with some input tensor.

When exporting to torchscript, is there a way to have 2 entry points:
One for  update()
One for predict()

Output:
Yes, you can define multiple entry points in a TorchScript model by using the @torch.jit.export decorator to specify which methods should be exported as entry points.
For example, given a PyTorch model defined as follows:
class MyModel(nn.Module):
def update(self):
    # Update some params.

def predict(self, X):
    # Predict with some input tensor.

You can use the @torch.jit.export decorator to specify that the update and predict methods should be exported as entry points in the resulting TorchScript module, like this:
class MyModel(nn.Module):
@torch.jit.export
def update(self):
    # Update some params.

@torch.jit.export
def predict(self, X):
    # Predict with some input tensor.

You can then export the MyModel class to TorchScript using the following code:
model = MyModel()
traced_model = torch.jit.script(model)

The resulting TorchScript module will have two entry points, update and predict, which you can use to call the corresponding methods of your model.
traced_model.update()
traced_model.predict(X)

Alternatively, you can also use the torch.jit.export decorator at the class level to specify that all of the methods in the class should be exported as entry points in the resulting TorchScript module. For example:
@torch.jit.export
class MyModel(nn.Module):
   def update(self):
       # Update some params.

   def predict(self, X):
       # Predict with some input tensor.

In this code, the @torch.jit.export decorator is applied to the MyModel class itself, which tells the torch.jit.script function to export all of the methods in the MyModel class as entry points in the resulting TorchScript module.
You can then export the MyModel class to TorchScript using the following code:
model = MyModel()
traced_model = torch.jit.script(model)

The resulting TorchScript module will have two entry points, update and predict, which you can use to call the corresponding methods of your model.
traced_model.update()
traced_model.predict(X)


Instruction: Pytorch lightning logger doesn't work as expected

Input:
Good evening,
I am a beginner in Pytorch lightning and I am trying to implement a NN and plot the graph (loss and accuracy) on various sets.
The code is this one
def training_step(self, train_batch, batch_idx):
    X, y = train_batch
    y_copy = y # Integer y for the accuracy
    X = X.type(torch.float32)
    y = y.type(torch.float32)
    # forward pass
    y_pred = self.forward(X).squeeze()
    # accuracy
    accuracy = Accuracy()
    acc = accuracy(y_pred, y_copy)
    # compute loss
    loss = self.loss_fun(y_pred, y)
    self.log_dict({'train_loss': loss, 'train_accuracy': acc}, on_step=False, on_epoch=True, prog_bar=True, logger=True)
    return loss

def validation_step(self, validation_batch, batch_idx):
    X, y = validation_batch
    X = X.type(torch.float32)
    # forward pass
    y_pred = self.forward(X).squeeze()
    # compute metrics
    accuracy = Accuracy()
    acc = accuracy(y_pred, y)
    loss = self.loss_fun(y_pred, y)
    self.log_dict({'validation_loss': loss, 'validation_accuracy': acc}, on_step=False, on_epoch=True, prog_bar=True, logger=True)
    return loss

def test_step(self, test_batch, batch_idx):
    X, y = test_batch
    X = X.type(torch.float32)
    # forward pass
    y_pred = self.forward(X).squeeze()
    # compute metrics
    accuracy = Accuracy()
    acc = accuracy(y_pred, y)
    loss = self.loss_fun(y_pred, y)
    self.log_dict({'test_loss': loss, 'test_accuracy': acc}, on_epoch=True, prog_bar=True, logger=True)
    return loss

After training the NN, I run this peace of code:
metrics = pd.read_csv(f"{trainer.logger.log_dir}/metrics.csv")
del metrics["step"]
metrics

and I obtain this

It is okay than on the validation set there's only one accuracy and one loss, because I am performing an hold out CV. On the test set, I noticed that the value test_accuracy=0.97 is the mean of all the accuracy for each epoch. with that I can't see the intermediate values (for each epoch) and then I can't plot any curve. It would be useful also when I'll do a cross validation with KFold.
Why he's taking the mean and How can I see the intermediate results ? For the training_step it works properly, I can't figure out why the logger doesn't perform the same print for the test_step.
Can someone help me please ?

Output:
It looks like the test_step method is logging the metrics using the on_epoch parameter, which means that the logged values will be averaged over the entire epoch and only logged once per epoch. To log the metrics at each step, you should set the on_epoch parameter to False in the test_step method like this:
self.log_dict({'test_loss': loss, 'test_accuracy': acc}, on_epoch=False, prog_bar=True, logger=True)

This will log the loss and accuracy at each step in the test_step method, allowing you to see the intermediate values and plot the curve.


Instruction: PyTorch doesn't import after installing Anaconda

Input:
I just installed PyTorch after installing Anaconda, and when I run iPython using Anaconda Python, it won't find PyTorch.

However I can verify PyTorch is in the pkgs directory of my anaconda folder.
What's going on?

Output: While the framework name is Pytorch, it must be imported using the torch namespace like so import torch. If you look at the official documentation here, you will see any pytorch related module import like nn or autograd is done through the torch namespace. I admit it is a bit confusing but I guess it was done to instill a sense of familiarity for old torch7 users. Also, there is a very good introduction tutorial which you could probably use to familiarize yourself better with the ins and outs of pytorch found here. Hope this helps!


Instruction: Explanation behind actor-critic algorithm in pytorch example?

Input:
Pytorch provides a good example of using actor-critic to play Cartpole in the OpenAI gym environment.

I'm confused about several of their equations in the code snippet found at https://github.com/pytorch/examples/blob/master/reinforcement_learning/actor_critic.py#L67-L79:

saved_actions = model.saved_actions
value_loss = 0
rewards = []
for r in model.rewards[::-1]:
    R = r + args.gamma * R
    rewards.insert(0, R)
rewards = torch.Tensor(rewards)
rewards = (rewards - rewards.mean()) / (rewards.std() + np.finfo(np.float32).eps)
for (action, value), r in zip(saved_actions, rewards):
    action.reinforce(r - value.data.squeeze())
    value_loss += F.smooth_l1_loss(value, Variable(torch.Tensor([r])))
optimizer.zero_grad()
final_nodes = [value_loss] + list(map(lambda p: p.action, saved_actions))
gradients = [torch.ones(1)] + [None] * len(saved_actions)
autograd.backward(final_nodes, gradients)
optimizer.step()

What do r and value mean in this case? Why do they run REINFORCE on the action space with the reward equal to r - value? And why do they try to set the value so that it matches r?

Thanks for your help!

Output:
First the rewards a collected for a time, along with the state:action that resulted in the reward
Then r - value is the difference between the expected reward and actual
That difference is used to adjust the expected value of that action from that state

So if in state "middle", the expected reward for action "jump" was 10  and the actual reward was only 2, then the AI was off by -8 (2-10).  Reinforce means "adjust expectations".  So if we adjust them by half, we'll new expected reward is 10-(8 *.5), or 6.  meaning the AI really thought it would get 10 for that, but now it's less confident and thinks 6 is a better guess.   So if the AI is not off by much, 10 - ( 2 *.5) = 9, it will adjust by a smaller amount.


Instruction: Pytorch and Polynomial Linear Regression issue

Input:
I have modified the code hat I found on the Pytorch github to suit my data, but my loss results are huge and with each iteration they get bigger and later become nan.Code doesn't give me any errors, just nor loss results and no predictions.
I have another code that deals withe the simple Linear Regression and all works fine. I guess I'm missing something simple here, but I'm unable to see it. any help would be appreciated.
Code:
import sklearn.linear_model as lm
from sklearn.preprocessing import PolynomialFeatures
import torch
import torch.autograd
import torch.nn.functional as F
from torch.autograd import Variable

train_data = torch.Tensor([
   [40,  6,  4],
   [44, 10,  4],
   [46, 12,  5],
   [48, 14,  7],
   [52, 16,  9],
   [58, 18, 12],
   [60, 22, 14],
   [68, 24, 20],
   [74, 26, 21],
   [80, 32, 24]])
test_data = torch.Tensor([
    [6, 4],
    [10, 5],
    [4, 8]])

x_train = train_data[:,1:3]
y_train = train_data[:,0]

POLY_DEGREE = 3
input_size = 2
output_size = 1

poly = PolynomialFeatures(input_size * POLY_DEGREE, include_bias=False)
x_train_poly = poly.fit_transform(x_train.numpy())

class Model(torch.nn.Module):

    def __init__(self):
        super(Model, self).__init__()
        self.fc = torch.nn.Linear(poly.n_output_features_, output_size)

    def forward(self, x):
        return self.fc(x)

model = Model()
criterion = torch.nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.001)

losses = []

for i in range(10):
    optimizer.zero_grad()
    outputs = model(Variable(torch.Tensor(x_train_poly)))
    print(outputs)
    loss = criterion(outputs, Variable(y_train))
    print(loss.data[0])
    losses.append(loss.data[0])
    loss.backward()
    optimizer.step()
    if loss.data[0] < 1e-4:
        break

print('n_iter', i)
print(loss.data[0])
plt.plot(losses)
plt.show()

output:

[393494300459008.0, inf, inf, inf, nan, nan, nan, nan, nan, nan]
n_iter
9 nan

Output:
There are a couple of things that contribute to the problem. Changing some or all of them will give you reasonable results and make learning possible.

Some of your (polynomial) features have a huge variance and are taking on very large values. Check out np.max(x_train_poly).When your weight matrix is randomly initialised, this causes the initial predictions to be largely off, and the loss to approach infinity quickly. To counteract this, you may want to standardise your features first (i.e. make mean 0 and variance 1 for each feature). Note, that in very deep networks a similar idea is used called "Batch Normalization". If you're interested, you can read more here: https://arxiv.org/abs/1502.03167 You can do the following to fix your example:

means = np.mean(x_train_poly,axis=0,keepdims=True)
std = np.std(x_train_poly,axis=0,keepdims=True)
x_train_poly = (x_train_poly - means) / std

Your current model, doesn't have any hidden layers, which is sort of the point of a neural network and building a non-linear regressor/ classifier. What you're doing right now is applying a linear transformation to the 27 input features to get something that is close to the output. You could add an additional layer like this:

hidden_dim = 50

class Model(torch.nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.layer1 = torch.nn.Linear(poly.n_output_features_, hidden_dim)
        self.layer2 = torch.nn.Linear(hidden_dim, output_size)

    def forward(self, x):
        return self.layer2(torch.nn.ReLU()(self.layer1(x)))

Note that I have added a non-linearity after the first linear transformation, because else there's no point having multiple layers.
The problem of initial predictions that are greatly off in the beginning and lead to the loss approaching infinity. You're using squared loss which essentially doubles the order of magnitude of your initial "mistake" in the loss function. And once the loss is infinity, you'll be unable to escape, because the gradient updates are essentially also infinity as you're using squared loss. An easy fix that is sometimes useful is to use the smooth L1 loss instead. Essentially MSE on the interval [0, 1] and L1 loss outside that interval. Change the following:

criterion = torch.nn.SmoothL1Loss()

That already gets you to something sensible (i.e. no infs anymore), but now
consider tuning the learning rate and introducing weight_decay. You may also want to change the optimizer. Some suggestions that works alright:

optimizer = torch.optim.SGD(model.parameters(), lr=0.01, weight_decay=1)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01, weight_decay=0.1)


Instruction: undefined symbol: PySlice_AdjustIndices when importing PyTorch

Input:
I am trying to use PyTorch, and I think there is some version of something that isn't lining up.

From what little I can suss out, it seems that there are some functions in the newest version of PyTorch (?) that can't be accessed on my system. I suspect it has something to do with Python version 3.6.1 as opposed to 3.6.0. But I can't figure it out. If anyone has any advice on what I can do to rectify this error:

/home/ubuntu/nbs/torch_utils.py in <module>()
----> 1 import torch
      2 import torch.nn as nn
      3 import torch.nn.parallel
      4 import torch.utils.data
      5 from torch import optim

/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/__init__.py in <module>()
     51 sys.setdlopenflags(_dl_flags.RTLD_GLOBAL | _dl_flags.RTLD_NOW)
     52
---> 53 from torch._C import *
     54
     55 __all__ += [name for name in dir(_C)

ImportError: /home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so: undefined symbol: PySlice_AdjustIndices

Here's the details on my system:

I am running Ubuntu on AWS, Ubuntu 16.04.2 LTS

my Cuda info is Cuda compilation tools, release 8.0, V8.0.61

I have Anaconda, conda 4.3.15

My python version is Python 3.6.0 :: Anaconda custom (64-bit)

Thanks.

Output:
I have the same problem, maybe the build is broken for Ubuntu / Python 3.6.

Anyway, until they fix this problem, you can install PyTorch by downgrading one version:

conda install pytorch=0.1.10 torchvision -c soumith

This version runs just fine on all my tests.


Instruction: Pytorch .backward() method without CUDA

Input:
I am trying to run the code in the Pytorch tutorial on the autograd module. However, when I run the .backwards() call, I get the error:

cuda runtime error (38) : no CUDA-capable device is detected at torch/csrc/autograd/engine.cpp:359

I admittedly have no CUDA-capable device set up at the moment, but it was my understanding that this wasn't strictly necessary (at least I didn't find it specified anywhere in the tutorial). So I was wondering if there is a way to still run the code without a CUDA-enabled GPU.

Output: You should transfer you network, inputs, and labels onto the cpu using: net.cpu(), Variable(inputs.cpu()), Variable(labels.cpu())


Instruction: about torch.nn.CrossEntropyLoss parameter shape

Input:
i'm learning pytorch, and taking the anpr project,which is based tensorflow
(https://github.com/matthewearl/deep-anpr,
http://matthewearl.github.io/2016/05/06/cnn-anpr/)
as a exercise, transplant it to pytorch platform.

there is a problem,i'm using nn.CrossEntropyLoss() as loss function:

criterion=nn.CrossEntropyLoss()

the output.data of model is:

  - 1.00000e-02 *
   - 2.5552 2.7582 2.5368 ... 5.6184 1.2288 -0.0076
   - 0.7033 1.3167 -1.0966 ... 4.7249 1.3217 1.8367
   - 0.7592 1.4777 1.8095 ... 0.8733 1.2417 1.1521
   - 0.1040 -0.7054 -3.4862 ... 4.7703 2.9595 1.4263
   - [torch.FloatTensor of size 4x253]

and targets.data is:

 - 1 0 0 ... 0 0 0
 - 1 0 0 ... 0 0 0
 - 1 0 0 ... 0 0 0
 - 1 0 0 ... 0 0 0
 - [torch.DoubleTensor of size 4x253]

when i call:

loss=criterion(output,targets)

error occured,information is:

TypeError: FloatClassNLLCriterion_updateOutput received an invalid combination of arguments - got (int, torch.FloatTensor, **torch.DoubleTensor**, torch.FloatTensor, bool, NoneType, torch.FloatTensor), but expected (int state, torch.FloatTensor input, **torch.LongTensor** target, torch.FloatTensor output, bool sizeAverage, [torch.FloatTensor weights or None], torch.FloatTensor total_weight)

'expected torch.LongTensor'......'got torch.DoubleTensor',but if i convert the targets into LongTensor:
torch.LongTensor(numpy.array(targets.data.numpy(),numpy.long))

call loss=criterion(output,targets), the error is:

RuntimeError: multi-target not supported at /data/users/soumith/miniconda2/conda-bld/pytorch-0.1.10_1488752595704/work/torch/lib/THNN/generic/ClassNLLCriterion.c:20

my last exercise is mnist, a example from pytorch,i made a bit modification,batch_size is 4,the loss function:

loss = F.nll_loss(outputs, labels)

outputs.data:

 - -2.3220 -2.1229 -2.3395 -2.3391 -2.5270 -2.3269 -2.1055 -2.2321 -2.4943 -2.2996

   -2.3653 -2.2034 -2.4437 -2.2708 -2.5114 -2.3286 -2.1921 -2.1771 -2.3343 -2.2533

   -2.2809 -2.2119 -2.3872 -2.2190 -2.4610 -2.2946 -2.2053 -2.3192 -2.3674 -2.3100

   -2.3715 -2.1455 -2.4199 -2.4177 -2.4565 -2.2812 -2.2467 -2.1144 -2.3321 -2.3009

   [torch.FloatTensor of size 4x10]

labels.data:

 - 8
 - 6
 - 0
 - 1
 - [torch.LongTensor of size 4]

the labels, for a input image,must be a single element, in upper example, there is 253 numbers, and in 'mnist',there is only one number, the shape of outputs is difference from labels.

i review the tensorflow manual, tf.nn.softmax_cross_entropy_with_logits,
'Logits and labels must have the sameshape [batch_size, num_classes] and the same dtype (either float32 or float64).'

does pytorch support the same function in tensorflow?

many thks

Output:
You can convert the targets that you have to a categorical representation.
In the example that you provide, you would have 1 0 0 0.. 0 if the class is 0, 0 1 0 0 ... if the class is 1, 0 0 1 0 0 0... if the class is 2 etc.
One quick way that I can think of is first convert the target Tensor to a numpy array, then convert it from one hot to a categorical array, and convert it back to a pytorch Tensor. Something like this:

targetnp=targets.numpy()
idxs=np.where(targetnp>0)[1]
new_targets=torch.LongTensor(idxs)
loss=criterion(output,new_targets)


Instruction: Trying to load a custom dataset in Pytorch

Input:
I'm just starting out with PyTorch and am, unfortunately, a bit confused when it comes to using my own training/testing image dataset for a custom algorithm. For starters, I am making a small "hello world"-esque convolutional shirt/sock/pants classifying network. I've only loaded a few images and am just making sure that PyTorch can load them and transform them down properly to 32x32 usable images. My ImageFolder is set up like so:

  imgs/socks/sockimages.jpeg
  imgs/pants/pantsimages.jpeg
  imgs/shirt/shirtimages.jpeg

and a similar setup for my testing images folder. According to my current knowledge, the image loader built into PyTorch should read the labels from the subfolder names within the training/test images. However, I'm getting a TypeError complaining that my iterator is not iterable. Here's my code and the error:

import torch
import torchvision
import torchvision.datasets as dset
import torchvision.transforms as transforms

transform = transforms.Compose(
[transforms.ToTensor(),
 transforms.Scale((32,32)),
 transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

trainset = dset.ImageFolder(root="imgs",transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,shuffle=True,         num_workers=2)

testset = dset.ImageFolder(root='tests',transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=4,shuffle=True,     num_workers=2)

classes=('shirt','pants','sock')

import matplotlib.pyplot as plt
import numpy as np

# functions to show an image
def imshow(img):
    img = img / 2 + 0.5     # unnormalize
    npimg = img.numpy()
    plt.imshow(np.transpose(npimg, (1, 2, 0)))

# get some random training images
dataiter = iter(trainloader)
images, labels = dataiter.next()

# show images
imshow(torchvision.utils.make_grid(images))
# print labels
print(' '.join('%5s' % classes[labels[j]] for j in range(4)))

Error:

TypeError: 'builtin_function_or_method' object is not iterable

It says that it is in reference to the line containing dataiter.next(), meaning that the compiler believes that I cannot iterate dataiter?

Please help! Thanks in advance,

-David Sillman, new to PyTorch

Output:
I think the error comes because in the transform.Compose you are doing first .ToTensor() and, instead, you should do .Scale(). Pytorch has transformations on tensors and on PIL Images, that are not interchangeable.
Reading the docs it says

  class torchvision.transforms.Scale(size, interpolation=2) [...]
   Rescale the input PIL.Image to the given size.

While you are changing that image to a Pytorch tensor before scaling thus making it crash.

It should be changed to:

transform = transforms.Compose(
                   [transforms.Scale((32,32)),
                    transforms.ToTensor(),
                    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

You will get this error when applying PIL Image transformations on tensors.


Instruction: LSTM time sequence generation using PyTorch

Input:
For several days now, I am trying to build a simple sine-wave sequence generation using LSTM, without any glimpse of success so far.

I started from the time sequence prediction example

All what I wanted to do differently is:

Use different optimizers (e.g RMSprob) than LBFGS
Try different signals (more sine-wave components)

This is the link to my code. "experiment.py" is the main file

What I do is:

I generate artificial time-series data (sine waves)
I cut those time-series data into small sequences
The input to my model is a sequence of time 0...T, and the output is a sequence of time 1...T+1

What happens is:

The training and the validation losses goes down smoothly
The test loss is very low
However, when I try to generate arbitrary-length sequences, starting from a seed (a random sequence from the test data), everything goes wrong. The output always flats out

I simply don't see what the problem is. I am playing with this for a week now, with no progress in sight.
I would be very grateful for any help.

Thank you

Output:
This is normal behaviour and happens because your network is too confident of the quality of the input and doesn't learn to rely on the past (on it's internal state) enough, relying soley on the input. When you apply the network to its own output in the generation setting, the input to the network is not as reliable as it was in the training or validation case where it got the true input.

I have two possible solutions for you:

The first is the simplest but less intuitive one: Add a little bit of Gaussian noise to your input. This will force the network to rely more on its hidden state.
The second, is the most obvious solution: during training, feed it not the true input but its generated output with a certain probability p. Start out training with p=0 and gradually increase it so that it learns to general longer and longer sequences, independently. This is called schedualed sampling, and you can read more about it here: https://arxiv.org/abs/1506.03099 .


Instruction: Default dilation value in PyTorch

Input:
As given in the documentation of PyTorch, the layer Conv2d uses a default dilation of 1. Does this mean that if I want to create a simple conv2d layer  I would have to write

nn.conv2d(in_channels = 3, out_channels = 64, kernel_size = 3, dilation = 0)

instead of simply writing

nn.conv2d(in_channels = 3, out_channels = 64, kernel_size = 3)

Or is it the case that in PyTorch dilation = 1 means same as dilation = 0 as given here in the Dilated Convolution section?

Output:
From the calculation of H_out, W_out in the documentation of pytorch, we can know that dilation=n means to make a pixel (1x1) of kernel to be nxn, where the original kernel pixel is at the topleft, and the rest pixels are empty (or filled with 0).

Thus dilation=1 is equivalent to the standard convolution with no dilation.


Instruction: How to train a reverse embedding, like vec2word?

Input:
how do you train a neural network to map from a vector representation, to one hot vectors? The example I'm interested in is where the vector representation is the output of a word2vec embedding, and I'd like to map onto the the individual words which were in the language used to train the embedding, so I guess this is vec2word?

In a bit more detail; if I understand correctly, a cluster of points in embedded space represents similar words. Thus if you sample from points in that cluster, and use it as the input to vec2word, the output should be a mapping to similar individual words?

I guess I could do something similar to an encoder-decoder, but does it have to be that complicated/use so many parameters?

There's this TensorFlow tutorial, how to train word2vec, but I can't find any help to do the reverse? I'm happy to do it using any deeplearning library, and it's OK to do it using sampling/probabilistic.

Thanks a lot for your help, Ajay.

Output: One easiest thing that you can do is to use the nearest neighbor word. Given a query feature of an unknown word fq, and a reference feature set of known words R={fr}, then you can find out what is the nearest fr* for fq, and use the corresponding fr* word as fq's word.


Instruction: multi-variable linear regression with pytorch

Input:
I'm working on a linear regression problem with Pytorch.
I've had success with the single variable case, however when I perform multi-variable linear regression I get the following error. How should I perform linear regression with multiple variables?

  TypeError                                 Traceback (most recent call
  last)  in ()
        9     optimizer.zero_grad() #gradient
       10     outputs = model(inputs) #output
  ---> 11     loss = criterion(outputs,targets) #loss function
       12     loss.backward() #backward propogation
       13     optimizer.step() #1-step optimization(gradeint descent)

  /anaconda/envs/tensorflow/lib/python3.6/site-packages/torch/nn/modules/module.py
  in call(self, *input, **kwargs)
      204
      205     def call(self, *input, **kwargs):
  --> 206         result = self.forward(*input, **kwargs)
      207         for hook in self._forward_hooks.values():
      208             hook_result = hook(self, input, result)

  /anaconda/envs/tensorflow/lib/python3.6/site-packages/torch/nn/modules/loss.py
  in forward(self, input, target)
       22         _assert_no_grad(target)
       23         backend_fn = getattr(self._backend, type(self).name)
  ---> 24         return backend_fn(self.size_average)(input, target)
       25
       26

  /anaconda/envs/tensorflow/lib/python3.6/site-packages/torch/nn/_functions/thnn/auto.py
  in forward(self, input, target)
       39         output = input.new(1)
       40         getattr(self._backend, update_output.name)(self._backend.library_state, input, target,
  ---> 41                                                    output, *self.additional_args)
       42         return output
       43

  TypeError: FloatMSECriterion_updateOutput received an invalid
  combination of arguments - got (int, torch.FloatTensor,
  torch.DoubleTensor, torch.FloatTensor, bool), but expected (int state,
  torch.FloatTensor input, torch.FloatTensor target, torch.FloatTensor
  output, bool sizeAverage)

here is code

#import
import torch
import torch.nn as nn
import numpy as np
import matplotlib.pyplot as plt
from torch.autograd import Variable

#input_size = 1
input_size = 3
output_size = 1
num_epochs = 300
learning_rate = 0.002

#Data set
#x_train = np.array([[1.564],[2.11],[3.3],[5.4]], dtype=np.float32)
x_train = np.array([[73.,80.,75.],[93.,88.,93.],[89.,91.,90.],[96.,98.,100.],[73.,63.,70.]],dtype=np.float32)
#y_train = np.array([[8.0],[19.0],[25.0],[34.45]], dtype= np.float32)
y_train = np.array([[152.],[185.],[180.],[196.],[142.]])
print('x_train:\n',x_train)
print('y_train:\n',y_train)

class LinearRegression(nn.Module):
    def __init__(self,input_size,output_size):
        super(LinearRegression,self).__init__()
        self.linear = nn.Linear(input_size,output_size)

    def forward(self,x):
        out = self.linear(x) #Forward propogation
        return out

model = LinearRegression(input_size,output_size)

#Lost and Optimizer
criterion = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(),lr=learning_rate)

#train the Model
for epoch in range(num_epochs):
    #convert numpy array to torch Variable
    inputs = Variable(torch.from_numpy(x_train)) #convert numpy array to torch tensor
    #inputs = Variable(torch.Tensor(x_train))
    targets = Variable(torch.from_numpy(y_train)) #convert numpy array to torch tensor

    #forward+ backward + optimize
    optimizer.zero_grad() #gradient
    outputs = model(inputs) #output
    loss = criterion(outputs,targets) #loss function
    loss.backward() #backward propogation
    optimizer.step() #1-step optimization(gradeint descent)

    if(epoch+1) %5 ==0:
        print('epoch [%d/%d], Loss: %.4f' % (epoch +1, num_epochs, loss.data[0]))
        predicted = model(Variable(torch.from_numpy(x_train))).data.numpy()
        plt.plot(x_train,y_train,'ro',label='Original Data')
        plt.plot(x_train,predicted,label='Fitted Line')
        plt.legend()
        plt.show()

Output:
You need to make sure that the data has the same type. In this case x_train is a 32 bit float while y_train is a Double. You have to use:

y_train = np.array([[152.],[185.],[180.],[196.],[142.]],dtype=np.float32)


Instruction: IOError: [Errno 28] No space left on device while installing pytorch

Input:
l got the following error when installing pytorch

 pip install http://download.pytorch.org/whl/cu80/torch-0.1.11.post5-cp27-none-linux_x86_64.whl
Collecting torch==0.1.11.post5 from http://download.pytorch.org/whl/cu80/torch-0.1.11.post5-cp27-none-linux_x86_64.whl
  Downloading http://download.pytorch.org/whl/cu80/torch-0.1.11.post5-cp27-none-linux_x86_64.whl (475.7MB)
100% |████████████████████████████████| 475.7MB 3.5MB/s

the following error

Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/local/lib/python2.7/dist-packages/pip/commands/install.py", line 335, in run
    wb.build(autobuilding=True)
  File "/usr/local/lib/python2.7/dist-packages/pip/wheel.py", line 749, in build
    self.requirement_set.prepare_files(self.finder)
  File "/usr/local/lib/python2.7/dist-packages/pip/req/req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/usr/local/lib/python2.7/dist-packages/pip/req/req_set.py", line 620, in _prepare_file
    session=self.session, hashes=hashes)
  File "/usr/local/lib/python2.7/dist-packages/pip/download.py", line 821, in unpack_url
    hashes=hashes
  File "/usr/local/lib/python2.7/dist-packages/pip/download.py", line 663, in unpack_http_url
    unpack_file(from_path, location, content_type, link)
  File "/usr/local/lib/python2.7/dist-packages/pip/utils/__init__.py", line 599, in unpack_file
    flatten=not filename.endswith('.whl')
  File "/usr/local/lib/python2.7/dist-packages/pip/utils/__init__.py", line 501, in unzip_file
    fp.write(data)
IOError: [Errno 28] No space left on device

What's wrong ?  l don't understand the error coming from /python2.7/dist-packages/pip/

Output:
In my case the system /tmp ran out of space, hence the error despite the installation path has enough space left.
export TMPDIR=/where/you/have/enough/space/left solves the problem for me.


Instruction: How can l uninstall PyTorch with Anaconda?

Input:
I installed PyTorch with:
conda install pytorch torchvision cuda80 -c soumith

How do I uninstall and remove all PyTorch dependencies?

Output:
From the anaconda docs, you can uninstall with conda uninstall

Try

conda uninstall pytorch torchvision cuda80 -c soumith

Alternatively, the pytorch docs suggest

conda uninstall pytorch
pip uninstall torch
pip uninstall torch # run this command twice


Instruction: Numpy outperforms Tensorflow and Pytorch with same hyperparameters

Input:
I have made 3 Neural Networks for Joel's FizzBuzz implementation, each in Numpy, TensorFlow and Pytorch. With the same hyperparameters and 1k epochs, my numpy script converges to 0.002 loss yet my pytorch and tensorflow is still jumping around 0.6. Could someone please help me figure out whats happening. I dont think Google and [Facebook + Nvidia] made something dump than Numpy just for GPU boost. My code below

Numpy

import numpy as np

input_size = 10
epochs = 1000
batches = 64
lr = 0.01

def sig(val):
    return 1 / (1 + np.exp(-val))

def sig_d(val):
    sig_val = sig(val)
    return sig_val * (1 - sig_val)

def binary_enc(num):
    ret = [int(i) for i in '{0:b}'.format(num)]
    return [0] * (input_size - len(ret)) + ret

def binary_dec(array):
    ret = 0
    for i in array:
        ret = ret * 2 + int(i)
    return ret

def training_test_gen(x, y):
    assert len(x) == len(y)
    indices = np.random.permutation(range(len(x)))
    split_size = int(0.9 * len(indices))
    trX = x[indices[:split_size]]
    trY = y[indices[:split_size]]
    teX = x[indices[split_size:]]
    teY = y[indices[split_size:]]
    return trX, trY, teX, teY

def x_y_gen():
    x = []
    y = []
    for i in range(1000):
        x.append(binary_enc(i))
        if i % 15 == 0:
            y.append([1, 0, 0, 0])
        elif i % 5 == 0:
            y.append([0, 1, 0, 0])
        elif i % 3 == 0:
            y.append([0, 0, 1, 0])
        else:
            y.append([0, 0, 0, 1])
    return training_test_gen(np.array(x), np.array(y))

def check_fizbuz(i):
    if i % 15 == 0:
        return 'fizbuz'
    elif i % 5 == 0:
        return 'buz'
    elif i % 3 == 0:
        return 'fiz'
    else:
        return 'number'

trX, trY, teX, teY = x_y_gen()

w1 = np.random.randn(10, 100)
w2 = np.random.randn(100, 4)

b1 = np.zeros((1, 100))
b2 = np.zeros((1, 4))

no_of_batches = int(len(trX) / batches)
for epoch in range(epochs):
    for batch in range(no_of_batches):
        # forward
        start = batch * batches
        end = start + batches
        x = trX[start:end]
        y = trY[start:end]
        a2 = x.dot(w1) + b1
        h2 = sig(a2)
        a3 = h2.dot(w2) + b2
        hyp = sig(a3)
        error = hyp - y
        loss = (error ** 2).mean()

        # backward
        outerror = error
        outgrad = outerror * sig_d(a3)
        outdelta = h2.T.dot(outgrad)
        outbiasdelta = np.ones([1, batches]).dot(outgrad)

        hiddenerror = outerror.dot(w2.T)
        hiddengrad = hiddenerror * sig_d(a2)
        hiddendelta = x.T.dot(hiddengrad)
        hiddenbiasdelta = np.ones([1, batches]).dot(hiddengrad)

        w1 -= hiddendelta * lr
        b1 -= hiddenbiasdelta * lr
        w2 -= outdelta * lr
        b2 -= outbiasdelta * lr
    print(epoch, loss)

# test
a2 = teX.dot(w1) + b1
h2 = sig(a2)
a3 = h2.dot(w2) + b2
hyp = sig(a3)
outli = ['fizbuz', 'buz', 'fiz', 'number']
for i in range(len(teX)):
    num = binary_dec(teX[i])
    print(
        'Number: {} -- Actual: {} -- Prediction: {}'.format(
            num, check_fizbuz(num), outli[hyp[i].argmax()]))
print('Test loss: ', np.mean(teY - hyp))

Pytorch

import numpy as np
import torch as th
from torch.autograd import Variable

input_size = 10
epochs = 1000
batches = 64
lr = 0.01

def binary_enc(num):
    ret = [int(i) for i in '{0:b}'.format(num)]
    return [0] * (input_size - len(ret)) + ret

def binary_dec(array):
    ret = 0
    for i in array:
        ret = ret * 2 + int(i)
    return ret

def training_test_gen(x, y):
    assert len(x) == len(y)
    indices = np.random.permutation(range(len(x)))
    split_size = int(0.9 * len(indices))
    trX = x[indices[:split_size]]
    trY = y[indices[:split_size]]
    teX = x[indices[split_size:]]
    teY = y[indices[split_size:]]
    return trX, trY, teX, teY

def x_y_gen():
    x = []
    y = []
    for i in range(1000):
        x.append(binary_enc(i))
        if i % 15 == 0:
            y.append([1, 0, 0, 0])
        elif i % 5 == 0:
            y.append([0, 1, 0, 0])
        elif i % 3 == 0:
            y.append([0, 0, 1, 0])
        else:
            y.append([0, 0, 0, 1])
    return training_test_gen(np.array(x), np.array(y))

def check_fizbuz(i):
    if i % 15 == 0:
        return 'fizbuz'
    elif i % 5 == 0:
        return 'buz'
    elif i % 3 == 0:
        return 'fiz'
    else:
        return 'number'

trX, trY, teX, teY = x_y_gen()
if th.cuda.is_available():
    dtype = th.cuda.FloatTensor
else:
    dtype = th.FloatTensor
x = Variable(th.from_numpy(trX).type(dtype), requires_grad=False)
y = Variable(th.from_numpy(trY).type(dtype), requires_grad=False)

w1 = Variable(th.randn(10, 100).type(dtype), requires_grad=True)
w2 = Variable(th.randn(100, 4).type(dtype), requires_grad=True)

b1 = Variable(th.zeros(1, 100).type(dtype), requires_grad=True)
b2 = Variable(th.zeros(1, 4).type(dtype), requires_grad=True)

no_of_batches = int(len(trX) / batches)
for epoch in range(epochs):
    for batch in range(no_of_batches):
        start = batch * batches
        end = start + batches
        x_ = x[start:end]
        y_ = y[start:end]

        a2 = x_.mm(w1)
        a2 = a2.add(b1.expand_as(a2))
        h2 = a2.sigmoid()

        a3 = h2.mm(w2)
        a3 = a3.add(b2.expand_as(a3))
        hyp = a3.sigmoid()

        error = hyp - y_
        loss = error.pow(2).sum()
        loss.backward()

        w1.data -= lr * w1.grad.data
        w2.data -= lr * w2.grad.data
        b1.data -= lr * b1.grad.data
        b2.data -= lr * b2.grad.data
        w1.grad.data.zero_()
        w2.grad.data.zero_()
    print(epoch, error.mean().data[0])

TensorFlow

import tensorflow as tf
import numpy as np

input_size = 10
epochs = 1000
batches = 64
learning_rate = 0.01

def binary_enc(num):
    ret = [int(i) for i in '{0:b}'.format(num)]
    return [0] * (input_size - len(ret)) + ret

def binary_dec(array):
    ret = 0
    for i in array:
        ret = ret * 2 + int(i)
    return ret

def training_test_gen(x, y):
    assert len(x) == len(y)
    indices = np.random.permutation(range(len(x)))
    split_size = int(0.9 * len(indices))
    trX = x[indices[:split_size]]
    trY = y[indices[:split_size]]
    teX = x[indices[split_size:]]
    teY = y[indices[split_size:]]
    return trX, trY, teX, teY

def x_y_gen():
    x = []
    y = []
    for i in range(1000):
        x.append(binary_enc(i))
        if i % 15 == 0:
            y.append([1, 0, 0, 0])
        elif i % 5 == 0:
            y.append([0, 1, 0, 0])
        elif i % 3 == 0:
            y.append([0, 0, 1, 0])
        else:
            y.append([0, 0, 0, 1])
    return training_test_gen(np.array(x), np.array(y))

def check_fizbuz(i):
    if i % 15 == 0:
        return 'fizbuz'
    elif i % 5 == 0:
        return 'buz'
    elif i % 3 == 0:
        return 'fiz'
    else:
        return 'number'

trX, trY, teX, teY = x_y_gen()

x = tf.placeholder(tf.float32, [None, 10], name='x')
y = tf.placeholder(tf.float32, [None, 4], name='y')

lr = tf.placeholder(tf.float32, [], name='lr')

w1 = tf.Variable(tf.truncated_normal([10, 100]))
w2 = tf.Variable(tf.truncated_normal([100, 4]))

b1 = tf.Variable(tf.zeros(100))
b2 = tf.Variable(tf.zeros(4))

a2 = tf.sigmoid(tf.add(tf.matmul(x, w1), b1))
hyp = tf.sigmoid(tf.add(tf.matmul(a2, w2), b2))

cost = tf.reduce_mean(tf.square(hyp - y))
optmizer = tf.train.GradientDescentOptimizer(lr).minimize(cost)

prediction = tf.argmax(hyp, 1)

no_of_batches = int(len(trX) / batches)
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    for epoch in range(epochs):
        p = np.random.permutation(range(len(trX)))
        trX = trX[p]
        trY = trY[p]
        for batch in range(no_of_batches):
            start = batch * batches
            end = start + batches
            input_batch = trX[start: end]
            target_batch = trY[start: end]
            sess.run(
                optmizer, feed_dict={x: input_batch, y: target_batch, lr: learning_rate})
            if epoch % 100 == 0:
                a = np.argmax(teY, axis=1)
                b = sess.run(prediction, feed_dict={x: teX})
                acc = np.mean(a == b)
                out_cost = sess.run(
                    cost, feed_dict={x: input_batch, y: target_batch, lr: learning_rate})
                print('cost - {} --- accuracy - {}'.format(out_cost.mean(), acc))

Output:
Can't say I've checked all your code (that's too much work), but I noticed that you are not zeroing your bias gradients in PyTorch, so these gradients will keep growing, killing the optimization algorithm. (There may be other problems)

The more idiomatic way to write this would be:

optimizer = torch.optim.SGD(net.parameters(), lr)

for _ in range(steps):

    optimizer.zero_grad()

    # ...

As a general note, it shouldn't matter which framework you use here: numbers are numbers: if you feed the same batches, and initialize the weights the same way, you should get approximately the same gradients.

Edit

Here's another big difference: in NP, you are calculating the mean error (across batches and channels), while in PT, you sum them. That's a 40-fold difference that will affect both the loss and the gradient.


Instruction: Standard deviation in batches

Input: I am using this function to compute the standard deviation of my output. I am evaluating my model on 2000 examples, in batches of 4. Can I just take the standard deviation per batch and then divide by 2000 ?

Output: No, you can't. Have a look at this post where batch updates of the mean and the standard deviation are explained in a nice way, with both math and code.


Instruction: How to set different learning rate for different layer in pytorch?

Input:
I'm doing fine-tuning with pytorch using resnet50 and want to set the learning rate of the last fully connected layer to 10^-3 while the learning rate of other layers be set to 10^-6. I know that I can just follow the method in its document:
optim.SGD([{'params': model.base.parameters()},
           {'params': model.classifier.parameters(), 'lr': 1e-3}],
          lr=1e-2, momentum=0.9)

But is there anyway that I do not need to set the parameters layer by layer

Output:
You can group layers. If you want to group all linear layers, the best way to do it is use modules:
param_grp = []

for idx, m in enumerate(model.modules()):
    if isinstance(m, nn.Linear):
        param_grp.append(m.weight)


Instruction: PyTorch network produces constant output

Input:
I am trying to train a simple MLP to approximate y=f(a,b,c).
My code is as below.
import torch
import torch.nn as nn
from torch.autograd import Variable

# hyper parameters
input_size = 3
output_size = 1
num_epochs = 50
learning_rate = 0.001

# Network definition
class FeedForwardNet(nn.Module):
    def __init__(self, l1_size, l2_size):
      super(FeedForwardNet, self).__init__()
      self.fc1 = nn.Linear(input_size, l1_size)
      self.relu1 = nn.ReLU()
      self.fc2 = nn.Linear(l1_size, l2_size)
      self.relu2 = nn.ReLU()
      self.fc3 = nn.Linear(l2_size, output_size)

    def forward(self, x):
      out = self.fc1(x)
      out = self.relu1(out)
      out = self.fc2(out)
      out = self.relu2(out)
      out = self.fc3(out)
      return out

model = FeedForwardNet(5 , 3)

# sgd optimizer
optimizer = torch.optim.SGD(model.parameters(), learning_rate, momentum=0.9)

for epoch in range(11):
    print ('Epoch ', epoch)
    for i in range(trainX_light.shape[0]):
    X = Variable( torch.from_numpy(trainX_light[i]).view(-1, 3) )
    Y = Variable( torch.from_numpy(trainY_light[i]).view(-1, 1) )
    # forward
    optimizer.zero_grad()
    output = model(X)

    loss = (Y - output).pow(2).sum()
    print (output.data[0,0])
    loss.backward()
    optimizer.step()
    totalnorm = 0
    for p in model.parameters():
        modulenorm = p.grad.data.norm()
        totalnorm += modulenorm ** 2
        totalnorm = math.sqrt(totalnorm)

     print (totalnorm)

    # validation code
    if (epoch + 1) % 5 == 0:
    print (' test points',testX_light.shape[0])
    total_loss = 0
    for t in range(testX_light.shape[0]):
        X = Variable( torch.from_numpy(testX_light[t]).view(-1, 3) )
        Y = Variable( torch.from_numpy(testY_light[t]).view(-1, 1) )
        output = model(X)
        loss = (Y - output).pow(2).sum()
        print (output.data[0,0])
        total_loss += loss
    print ('epoch ', epoch, 'avg_loss ', total_loss.data[0] / testX_light.shape[0])

print ('Done')

The problem that I have now is, the validation code

output = model(X)

is always producing an exact same output value (I guess this value is some sort of garbage). I am not sure what mistake I am doing in this part. Could some help me figure out the mistake in my code?

Output: The reason that network produced random values (and inf later) was the exploding gradient problem. Clipping the gradient (torch.nn.utils.clip_grad_norm(model.parameters(), 0.1)) helped.


Instruction: Training speed on GPU become slower overtime

Input:
My model training speed becomes slower over time. Every epoch take longer time to train.
Here is the full source code with my preprocess sentiment tree bank data (put glove.840B.300d.txt into data/glove).
Install some python packages:
pip install meowlogtool
pip install tqdm

Command to run:
python sentiment.py --emblr 0 --rel_dim 0 --tag_dim 0 --optim adagrad --name basic --lr 0.05 --wd 1e-4 --at_hid_dim 0

Model source code for you to read
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable as Var
import utils
import Constants
from model import SentimentModule
from embedding_model import EmbeddingModel

class SimpleGRU(nn.Module):
    """
    w[i] : (300, 1)
    h[i] : (150, 1)
    p[i] : (20, 1)
    r[i] : (20, 1)
    k[i] : (150, 1)
    x[i] : (20 + 150 + 300 + 20 = 490, 1) (490, 1)
    Uz, Ur, Uh : (150, 150) => 67500 => (450, 450)
    Wz, Wr, Wh : (150, 20 + 150 + 300 + 20) (150, 490)
    """
    def __init__(self, cuda, in_dim, hid_dim):
        super(SimpleGRU, self).__init__()
        self.cudaFlag = cuda

        self.Uz = nn.Linear(hid_dim, hid_dim)
        self.Ur = nn.Linear(hid_dim, hid_dim)
        self.Uh = nn.Linear(hid_dim, hid_dim)

        self.Wz = nn.Linear(in_dim, hid_dim)
        self.Wr = nn.Linear(in_dim, hid_dim)
        self.Wh = nn.Linear(in_dim, hid_dim)

        if self.cudaFlag:
            self.Uz = self.Uz.cuda()
            self.Ur = self.Uz.cuda()
            self.Uh = self.Uz.cuda()

            self.Wz = self.Wz.cuda()
            self.Wr = self.Wr.cuda()
            self.Wh = self.Wh.cuda()

    def forward(self, x, h_prev):
        """
    Simple-GRU(compress_x[v], h[t-1]) :
    z[t]         := s(Wz *compress_x[t]+ Uz * h[t-1] + bz)
    r[t]         := s(Wr * compress_x[t] + Ur * h[t-1] + br)
    h_temp[t]     := g(Wh * compress_x[t] + Uh * h[t-1] + bh)
    h[t]         := r[t] .* h[t-1] + (1 - z[t]) .* h_temp[t]
    return h[t]
        :param x: compress_x[t]
        :param h_prev: h[t-1]
        :return:
        """
        z = F.sigmoid(self.Wz(x) + self.Uz(h_prev))
        r = F.sigmoid(self.Wr(x) + self.Ur(h_prev))
        h_temp = F.tanh(self.Wh(x) + self.Uh(h_prev))
        h = r*h_prev + (1-z)*h_temp
        return h

class TreeSimpleGRU(nn.Module):
    def __init__(self, cuda, word_dim, tag_dim, rel_dim, mem_dim, at_hid_dim, criterion, leaf_h = None):
        super(TreeSimpleGRU, self).__init__()
        self.cudaFlag = cuda
        # self.gru_cell = nn.GRUCell(word_dim + tag_dim, mem_dim)
        self.gru_cell = SimpleGRU(self.cudaFlag, word_dim+tag_dim, mem_dim)
        self.gru_at = GRU_AT(self.cudaFlag, word_dim + tag_dim + rel_dim + mem_dim, at_hid_dim ,mem_dim)
        self.mem_dim = mem_dim
        self.in_dim = word_dim
        self.tag_dim = tag_dim
        self.rel_dim = rel_dim
        self.leaf_h = leaf_h # init h for leaf node
        if self.leaf_h == None:
            self.leaf_h = Var(torch.rand(1, self.mem_dim))
            torch.save(self.leaf_h, 'leaf_h.pth')

        if self.cudaFlag:
            self.leaf_h = self.leaf_h.cuda()

        self.criterion = criterion
        self.output_module = None

    def getParameters(self):
        """
        Get flatParameters
        note that getParameters and parameters is not equal in this case
        getParameters do not get parameters of output module
        :return: 1d tensor
        """
        params = []
        for m in [self.gru_cell, self.gru_at]:
            # we do not get param of output module
            l = list(m.parameters())
            params.extend(l)

        one_dim = [p.view(p.numel()) for p in params]
        params = F.torch.cat(one_dim)
        return params

    def set_output_module(self, output_module):
        self.output_module = output_module

    def forward(self, tree, w_emb, tag_emb, rel_emb, training = False):
        loss = Var(torch.zeros(1))  # init zero loss
        if self.cudaFlag:
            loss = loss.cuda()

        for idx in xrange(tree.num_children):
            _, child_loss = self.forward(tree.children[idx], w_emb, tag_emb, rel_emb, training)
            loss = loss + child_loss

        if tree.num_children > 0:
            child_rels, child_k  = self.get_child_state(tree, rel_emb)
            if self.tag_dim > 0:
                tree.state = self.node_forward(w_emb[tree.idx - 1], tag_emb[tree.idx -1], child_rels, child_k)
            else:
                tree.state = self.node_forward(w_emb[tree.idx - 1], None, child_rels, child_k)
        elif tree.num_children == 0:
            if self.tag_dim > 0:
                tree.state = self.leaf_forward(w_emb[tree.idx - 1], tag_emb[tree.idx -1])
            else:
                tree.state = self.leaf_forward(w_emb[tree.idx - 1], None)

        if self.output_module != None:
            output = self.output_module.forward(tree.state, training)
            tree.output = output
            if training and tree.gold_label != None:
                target = Var(utils.map_label_to_target_sentiment(tree.gold_label))
                if self.cudaFlag:
                    target = target.cuda()
                loss = loss + self.criterion(output, target)
        return tree.state, loss

    def leaf_forward(self, word_emb, tag_emb):
        """
        Forward function for leaf node
        :param word_emb:  word embedding of current node u
        :param tag_emb: tag embedding of current node u
        :return: k of current node u
        """
        h = self.leaf_h
        if self.cudaFlag:
            h = h.cuda()
        if self.tag_dim > 0:
            x = F.torch.cat([word_emb, tag_emb], 1)
        else:
            x = word_emb
        k = self.gru_cell(x, h)
        return k

    def node_forward(self, word_emb, tag_emb, child_rels, child_k):
        """
        Foward function for inner node
        :param word_emb: word embedding of current node u
        :param tag_emb: tag embedding of current node u
        :param child_rels (tensor): rels embedding of child node v
        :param child_k (tensor): k of child node v
        :return:
        """
        n_child = child_k.size(0)
        h = Var(torch.zeros(1, self.mem_dim))
        if self.cudaFlag:
            h = h.cuda()

        for i in range(0, n_child):
            k = child_k[i]
            x_list = [word_emb, k]
            if self.rel_dim >0:
                rel = child_rels[i]
                x_list.append(rel)
            if self.tag_dim > 0:
                x_list.append(tag_emb)
            x = F.torch.cat(x_list, 1)
            h = self.gru_at(x, h)
        k = h
        return k

    def get_child_state(self, tree, rels_emb):
        """
        Get child rels, get child k
        :param tree: tree we need to get child
        :param rels_emb (tensor):
        :return:
        """
        if tree.num_children == 0:
            assert False #  never get here
        else:
            child_k = Var(torch.Tensor(tree.num_children, 1, self.mem_dim))
            if self.rel_dim>0:
                child_rels = Var(torch.Tensor(tree.num_children, 1, self.rel_dim))
            else:
                child_rels = None
            if self.cudaFlag:
                child_k = child_k.cuda()
                if self.rel_dim > 0:
                    child_rels = child_rels.cuda()
            for idx in xrange(tree.num_children):
                child_k[idx] = tree.children[idx].state
                if self.rel_dim > 0:
                    child_rels[idx] = rels_emb[tree.children[idx].idx - 1]
        return child_rels, child_k

class AT(nn.Module):
    """
    AT(compress_x[v]) := sigmoid(Wa * tanh(Wb * compress_x[v] + bb) + ba)
    """
    def __init__(self, cuda, in_dim, hid_dim):
        super(AT, self).__init__()
        self.cudaFlag = cuda
        self.in_dim = in_dim
        self.hid_dim = hid_dim

        self.Wa = nn.Linear(hid_dim, 1)
        self.Wb = nn.Linear(in_dim, hid_dim)

        if self.cudaFlag:
            self.Wa = self.Wa.cuda()
            self.Wb = self.Wb.cuda()

    def forward(self, x):
        out = F.sigmoid(self.Wa(F.tanh(self.Wb(x))))
        return out

class GRU_AT(nn.Module):
    def __init__(self, cuda, in_dim, at_hid_dim ,mem_dim):
        super(GRU_AT, self).__init__()
        self.cudaFlag = cuda
        self.in_dim = in_dim
        self.mem_dim = mem_dim
        self.at_hid_dim = at_hid_dim
        if at_hid_dim > 0:
            self.at = AT(cuda, in_dim, at_hid_dim)
        self.gru_cell = SimpleGRU(self.cudaFlag, in_dim, mem_dim)

        if self.cudaFlag:
            if at_hid_dim > 0:
                self.at = self.at.cuda()
            self.gru_cell = self.gru_cell.cuda()

    def forward(self, x, h_prev):
        """

        :param x:
        :param h_prev:
        :return: a * m + (1 - a) * h[t-1]
        """
        m = self.gru_cell(x, h_prev)
        if self.at_hid_dim > 0:
            a = self.at.forward(x)
            h = torch.mm(a, m) + torch.mm((1-a), h_prev)
        else:
            h = m
        return h

class TreeGRUSentiment(nn.Module):
    def __init__(self, cuda, in_dim, tag_dim, rel_dim, mem_dim, at_hid_dim, num_classes, criterion):
        super(TreeGRUSentiment, self).__init__()
        self.cudaFlag = cuda
        self.tree_module = TreeSimpleGRU(cuda, in_dim, tag_dim, rel_dim, mem_dim, at_hid_dim, criterion)
        self.output_module = SentimentModule(cuda, mem_dim, num_classes, dropout=True)
        self.tree_module.set_output_module(self.output_module)

    def get_tree_parameters(self):
        return self.tree_module.getParameters()

    def forward(self, tree, sent_emb, tag_emb, rel_emb, training = False):
        # sent_emb = F.torch.unsqueeze(self.word_embedding.forward(sent_inputs), 1)
        # tag_emb = F.torch.unsqueeze(self.tag_emb.forward(tag_inputs), 1)
        # rel_emb = F.torch.unsqueeze(self.rel_emb.forward(rel_inputs), 1)
        # sent_emb, tag_emb, rel_emb = self.embedding_model(sent_inputs, tag_inputs, rel_inputs)

        tree_state, loss = self.tree_module(tree, sent_emb, tag_emb, rel_emb, training)
        output = tree.output
        return output, loss

Output:
Why does neural network learning slow down as the error gets lower?

The reasons for the slowdown are not fully understood, but we have some basic ideas.
For classifiers, most training examples start out as incorrectly classified. Over time, more of them become correctly classified. Early in learning, you might have a nearly 100% error rate, so every example in the minibatch contributes to learning. Late in learning, you might have nearly a 0% error rate, so almost none of the examples in the minibatch contribute to learning. This problem can be resolved to some extent by using hard example mining or importance sampling. Both of these are just techniques for training on more difficult examples more often.
There are other more complicated reasons. One of them is that the condition number of the Hessian tends to worsen a lot as learning progresses, so that the optimal step size becomes smaller and smaller.


Instruction: How to asynchronously load and train batches to train a DeepLearning model?

Input:
I have 3TB dataset and 64GB RAM and a 12 core CPU and one 12GB GPU. would like to train a deep learning model on this dataset. How do I have asynchronous load of batches and training of the model? I want to make sure disk load of data doesn't block training loop to be waiting for the new batch to load into memory.

I am not language dependent and the easiest library that can do this without friction wins but I prefer one of torch, pytorch, tensorflow.

Output:
We solved this problem in the way @mo-hossny described above (not "tied to the Imagenet folder structure") with Keras (tensorflow backend) and described it in gory detail here.

A brief summary of that: most ML tutorials show a directory structure where the class of training (and test) examples is implied by the subdirectory. For instance, you might see subdirectories and files like data/train/cats/???.png and data/train/dogs/???.png, etc.

If instead you create a simple Pandas DataFrame to hold the unique id, class label and file path for each train/test sample, then you can shuffle this DataFrame at the start of each epoch, loop over it in mini-batches and use a generator to send each chunk to the GPU. In the background, the CPU is keeping the queue of chunks full, standing by to send each subsequent one to the GPU as soon as it finishes its current batch.

An example of such a DataFrame is:

df

       object_id   bi  multi                                    path
index
 0        461756  dog  white    /path/to/imgs/756/61/blah_461756.png
 1       1161756  cat  black   /path/to/imgs/756/61/blah_1161756.png
 2       3303651  dog  white   /path/to/imgs/651/03/blah_3303651.png
 3       3367756  dog   grey   /path/to/imgs/756/67/blah_3367756.png
 4       3767756  dog   grey   /path/to/imgs/756/67/blah_3767756.png
 5       5467756  cat  black   /path/to/imgs/756/67/blah_5467756.png
 6       5561756  dog  white   /path/to/imgs/756/61/blah_5561756.png
 7      31255756  cat   grey  /path/to/imgs/756/55/blah_31255756.png
 8      35903651  cat  black  /path/to/imgs/651/03/blah_35903651.png
 9      44603651  dog  black  /path/to/imgs/651/03/blah_44603651.png
10      49557622  cat  black  /path/to/imgs/622/57/blah_49557622.png
11      58164756  dog   grey  /path/to/imgs/756/64/blah_58164756.png
12      95403651  cat  white  /path/to/imgs/651/03/blah_95403651.png
13      95555756  dog   grey  /path/to/imgs/756/55/blah_95555756.png

I've included labels for binomial and multinomial versions of the problem do demonstrate that the same DataFrame and files can be used in different classification settings.

Once you have this going, the Keras generator code is pretty short and sweet:

train_generator = generator_from_df(df, batch_size, target_size)

where df is similar to my example above and the function generator_from_df() is defined here. It simply loops through the df in chunks of a given size; reads, normalized and concatenates the pixel data specified in the chunk's rows; and finally yields (hence the generator) the X (pixels) and Y (labels) data. The heart of it is very similar to:

i, j = 0, batch_size
for _ in range(nbatches):
    sub = df.iloc[i:j]
    X = np.array([
        (2 *
         (img_to_array(load_img(f, target_size=target_size))
          / 255.0 - 0.5))
        for f in sub.imgpath])
    Y = sub.target.values
    yield X, Y
    i = j
    j += batch_size
    count += 1

Note the references and code in the post: we aggregated helpful hints from others in the Keras pages and here on Stackoverflow.


Instruction: PyTorch + CUDA 7.5 error

Input:
I have non-sudo access to a machine with NVIDIA GPUs and CUDA 7.5 installed. I installed PyTorch with CUDA 7.5 support, which seems to have worked:

>>> import torch
>>> torch.cuda.is_available()
True

To get some practice, I followed tutorial for machine translation using RNNs. When I set USE_CUDA = False and the CPUs are used, everything works quite alright. However, when want to utilize the GPUs with USE_CUDA = True I get the following error:

Traceback (most recent call last):
  ...
  File "seq2seq.py", line 229, in train
    encoder_output, encoder_hidden = encoder(input_variable[ei], encoder_hidden)
  File "/.../python2.7/site-packages/torch/nn/modules/module.py", line 206, in __call__
    result = self.forward(*input, **kwargs)
  File "seq2seq.py", line 144, in forward
    output, hidden = self.gru(embedded, hidden)
  File "/.../python2.7/site-packages/torch/nn/modules/module.py", line 206, in __call__
    result = self.forward(*input, **kwargs)
  File "/.../python2.7/site-packages/torch/nn/modules/rnn.py", line 91, in forward
    output, hidden = func(input, self.all_weights, hx)
  ...
  File "/.../python2.7/site-packages/torch/backends/cudnn/rnn.py", line 42, in init_rnn_descriptor
    cudnn.DropoutDescriptor(handle, dropout_p, fn.dropout_seed)
  File "/usr/lib/python2.7/ctypes/__init__.py", line 383, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: python: undefined symbol: cudnnCreateDropoutDescriptor
Exception AttributeError: 'python: undefined symbol: cudnnDestroyDropoutDescriptor' in <bound method DropoutDescriptor.__del__ of <torch.backends.cudnn.DropoutDescriptor object at 0x7fe540efec10>> ignored

I've tried to use Google to search for that error but got no meaningful results. Since I'm rather a newbie with PyTorch and CUDA, I have no idea how to go on from here. The full setup is Ubuntu 14.04, Python 2.7, CUDA 7.5.

Output:
As stated in the comments: your error is with outdated CUDNN, and can be resolved by upgrading.

Install current versions of CUDA, CUDNN, and PyTorch, then you'll be fine.


Instruction: Simple LSTM in PyTorch with Sequential module

Input:
In PyTorch, we can define architectures in multiple ways. Here, I'd like to create a simple LSTM network using the Sequential module.

In Lua's torch I would usually go with:

model = nn.Sequential()
model:add(nn.SplitTable(1,2))
model:add(nn.Sequencer(nn.LSTM(inputSize, hiddenSize)))
model:add(nn.SelectTable(-1)) -- last step of output sequence
model:add(nn.Linear(hiddenSize, classes_n))

However, in PyTorch, I don't find the equivalent of SelectTable to get the last output.

nn.Sequential(
  nn.LSTM(inputSize, hiddenSize, 1, batch_first=True),
  # what to put here to retrieve last output of LSTM ?,
  nn.Linear(hiddenSize, classe_n))

Output:
Define a class to extract the last cell output:
# LSTM() returns tuple of (tensor, (recurrent state))
class extract_tensor(nn.Module):
    def forward(self,x):
        # Output shape (batch, features, hidden)
        tensor, _ = x
        # Reshape shape (batch, hidden)
        return tensor[:, -1, :]

nn.Sequential(
    nn.LSTM(inputSize, hiddenSize, 1, batch_first=True),
    extract_tensor(),
    nn.Linear(hiddenSize, classe_n)
)


Instruction: Hyperparameter optimization for Pytorch model

Input: What is the best way to perform hyperparameter optimization for a Pytorch model? Implement e.g. Random Search myself? Use Skicit Learn? Or is there anything else I am not aware of?

Output:
Many researchers use RayTune. It's a scalable hyperparameter tuning framework, specifically for deep learning. You can easily use it with any deep learning framework (2 lines of code below), and it provides most state-of-the-art algorithms, including HyperBand, Population-based Training, Bayesian Optimization, and BOHB.
import torch.optim as optim
from ray import tune
from ray.tune.examples.mnist_pytorch import get_data_loaders, ConvNet, train, test

def train_mnist(config):
    train_loader, test_loader = get_data_loaders()
    model = ConvNet()
    optimizer = optim.SGD(model.parameters(), lr=config["lr"])
    for i in range(10):
        train(model, optimizer, train_loader)
        acc = test(model, test_loader)
        tune.report(mean_accuracy=acc)

analysis = tune.run(
    train_mnist, config={"lr": tune.grid_search([0.001, 0.01, 0.1])})

print("Best config: ", analysis.get_best_config(metric="mean_accuracy"))

# Get a dataframe for analyzing trial results.
df = analysis.dataframe()

[Disclaimer: I contribute actively to this project!]


Instruction: how tf.space_to_depth() works in tensorflow?

Input:
I am a pytorch user. I have got a pretrained model in tensorflow and I would like to transfer it into pytorch. In one part of model architecture, I mean in tensorflow-defined model, there is a function tf.space_to_depth which transfers an input size of (None, 38,38,64) to (None, 19,19, 256). (https://www.tensorflow.org/api_docs/python/tf/space_to_depth) is the doc of this function. But I could not understand what this function actually do. Could you please provide some numpy codes to illustrate it for me?

Actually I would like to make an exact similar layer in pytorch.

Some codes in tensorflow reveals another secret:
Here is some codes:

import numpy as np
import tensorflow as tf

norm = tf.random_normal([1, 2, 2, 1], mean=0, stddev=1)
trans = tf.space_to_depth(norm,2)

with tf.Session() as s:
    norm = s.run(norm)
    trans = s.run(trans)

print("Norm")
print(norm.shape)
for index,value in np.ndenumerate(norm):
    print(value)

print("Trans")
print(trans.shape)
for index,value in np.ndenumerate(trans):
    print(value)

And here is the output:

Norm
(1, 2, 2, 1)
0.695261
0.455764
1.04699
-0.237587
Trans
(1, 1, 1, 4)
1.01139
0.898777
0.210135
2.36742

As you can see above, In Addition to data reshaping, the tensor values has changed!

Output:
This tf.space_to_depth divides your input into blocs and concatenates them.

In your example the input is 38x38x64 (and I guess the block_size is 2). So the function divides your input into 4 (block_size x block_size) and concatenates them which gives your 19x19x256 output.

You just need to divide each of your channel (input) into block_size*block_size patches (each patch has a size of width/block_size x height/block_size) and concatenate all of these patches. Should be pretty straightforward with numpy.

Hope it helps.


Instruction: Long Sequence In a seq2seq model with attention?

Input:
I am following along this pytorch tutorial and trying to apply this principle to summarization, where the encoding sequence would be around 1000 words and decoder target 200 words.
How do I apply seq2seq to this? I know it would be very expensive and almost infeasible to run through the whole sequence of 1000 words at once. So dividing the seq into say 20 seq and running in parallel could be an answer. But I'm not sure how to implement it; I also want to incorporate attention into it.

Output:
You can not parallelize RNN in time (1000 here) because they are inherently sequential.

You can use a light RNN, something like QRNN or SRU as a faster alternative(which is still sequential).

Another common sequence processing modules are TCN and Transformers which are both parallelizable in time.

Also, note that all of them can be used with attention and work perfectly fine with text.


Instruction: Basic multi GPU parallelization of matrix multiplication

Input:
I want to parallelize the simple following expression on 2 GPUs: C =  A^n + B^n by calculating A^n on GPU 0 and B^n on GPU 1 before summing the results.

In TensorFlow I would go like:

with tf.device('/gpu:0'):
    An = matpow(A, n)
with tf.device('/gpu:1'):
    Bn = matpow(B, n)
with tf.Session() as sess:
    C = sess.run(An + Bn)

However, since PyTorch is dynamic, I'm having trouble doing the same thing. I tried the following but it only takes more time.

with torch.cuda.device(0):
    A = A.cuda()
with torch.cuda.device(1):
    B = B.cuda()
C = matpow(A, n) + matpow(B, n).cuda(0)

I know there is a module to parallelize models on the batch dimension using torch.nn.DataParallel but here I try to do something more basic.

Output:
You can use cuda streams for this. This will not necessarily distribute it over two devices, but the execution will be in parallel.

s1 = torch.cuda.Stream()
s2 = torch.cuda.Stream()

with torch.cuda.stream(s1):
    A = torch.pow(A,n)
with torch.cuda.stream(s2):
    B = torch.pow(B,n)

C = A+B

Although I'm not sure whether it will really speed up your computation if you only parallelize this one operation. Your matrices must be really big.

If your requirement is to split it across devices, you can add this before the streams:

A = A.cuda(0)
B = B.cuda(1)

Then after the power operation, you need to get them on the same device again, e.g. B = B.cuda(0). After that you can do the addition.


Instruction: OpenNMT issues with Pytorch: cPickle.UnpicklingError: invalid load key, ''

Input:
I am trying to run the OpenNMT project using the instruction from the link: http://forum.opennmt.net/t/text-summarization-on-gigaword-and-rouge-scoring/85/6

I am using Python 2.7 and installed pytorch from the github repository.
I am trying to run the program using the prebuild model of the OpenNMT, which I have downloaded from the following: http://opennmt.net/Models/

I tried the command:

python translate.py -model textsum_epoch7_14.69_release.t7 -src data/Giga/input.txt

Got the following error:

Traceback (most recent call last):
  File "translate.py", line 151, in <module>
    main()
  File "translate.py", line 70, in main
    translator = onmt.Translator(opt)
  File "/home/ubuntu/opennmt/onmt/Translator.py", line 21, in __init__
    checkpoint = torch.load(opt.model)
  File "/usr/local/lib/python2.7/dist-packages/torch/serialization.py", line 229, in load
    return _load(f, map_location, pickle_module)
  File "/usr/local/lib/python2.7/dist-packages/torch/serialization.py", line 367, in _load
    magic_number = pickle_module.load(f)
cPickle.UnpicklingError: invalid load key, ''.

Kindly let me know what I need to do so that I can use the model and check the library OpenNMT.

Output:
The model you downloaded is for the Lua version of OpenNMT.

If you are just a user of the project, I recommend you to use this version as it is the most supported and stable.


Instruction: Pytorch: Sparse Matrix multiplcation

Input:
Given:

self.A = torch.autograd.Variable(random_sparse(n = dim))
self.w = torch.autograd.Variable(torch.Tensor(np.random.normal(0,1,(dim,dim))))

Goal1:

torch.mm(self.A, self.w)

Goal2:

torch.spmm(self.A, self.w)

Result1:

TypeError: Type torch.sparse.FloatTensor doesn't implement stateless method addmm

Result2:

AttributeError: 'module' object has no attribute 'spmm'

My PyTorch version is 0.1.12_2 - would greatly appreciate possible solutions.

Output:
spmm has been moved from torch module to torch.sparse module. For official documentation please check this link. There is also a warning in the beginning of the documentation of torch.sparse module:

  This API is currently experimental and may change in the near future.


Instruction: Parallel way of applying function element-wise to a Pytorch CUDA Tensor

Input: Suppose I have a torch CUDA tensor and I want to apply some function like sin() but I have explicitly defined the function F. How can I use parallel computation to apply F in Pytorch.

Output:
I think currently, it is not possible to explicit parallelize a function on a CUDA-Tensor. A possible solution could be, you can define a Function like the for example the non-linear activation functions. So you can feed forward it through the Net and your function.

The drawback is, it probably don't work, because you have to define a CUDA-Function and have to recompile pytorch.


Instruction: PyTorch specific inspection issues in PyCharm

Input:
Has anyone been able to resolve PyTorch specific inspection issues in PyCharm? Previous posts for non-PyTorch related issues suggest upgrading PyCharm, but I'm currently at the latest version. One option is to of course disable certain inspections entirely, but I'd rather avoid that.

Example: torch.LongTensor(x) gives me "Unexpected argument...", whereas both call signatures (with and without x) are both supported.

Output:
I believe it's because torch.LongTensor has no __init__ method for pycharm to find.

According to this source that I found thanks to this SO post :

  Use __new__ when you need to control the creation of a new instance.
  Use __init__ when you need to control initialization of a new instance.

  __new__ is the first step of instance creation.  It's called first,
  and is responsible for returning a new instance of your class.  In
  contrast, __init__ doesn't return anything; it's only responsible for
  initializing the instance after it's been created.

  In general, you shouldn't need to override __new__ unless you're
  subclassing an immutable type like str, int, unicode or tuple.

Since Tensors are types, it makes sense to define only new and no init.

You can experiment this behavior by testing the following classes :

torch.LongTensor(1)  # Unexpected arguments

Produces the warning while the following doesn't.

class MyLongTensor(torch.LongTensor):
    def __init__(self, *args, **kwargs):
        pass

MyLongTensor(1)  # No error

To confirm that the absence of __init__ is the culprit try :

class Example(object):
    pass

Example(0)  # Unexpected arguments

To find out by yourself, use pycharm to Ctrl+click on LongTensor then _TensorBase and look at the defined methods.


Instruction: AttributeError in creating module

Input:
I am getting the following error.

  AttributeError: cannot assign module before Module.init() call

I have a class as follows.

class Classifier(nn.Module):

    def __init__(self, dictionary, embeddings_index, max_seq_length, args):
        self.embedding = EmbeddingLayer(len(dictionary), args.emsize, args.dropout)
        self.drop = nn.Dropout(args.dropout)

What I am doing wrong here? I am beginner in PyTorch, please help!

Output:
The first thing you should always do when you create a module is call its super constructor. So, your class should look like this:

class Classifier(nn.Module):

    def __init__(self, dictionary, embeddings_index, max_seq_length, args):
        super(Classifier, self).__init__()
        '''Rest of your code goes here.'''


Instruction: Stacking copies of an array/ a torch tensor efficiently?

Input:
I'm a Python/Pytorch user. First, in numpy, let's say I have an array M of size LxL, and i want to have the following
array: A=(M,...,M) of size, say, NxLxL, is there a more elegant/memory efficient way of doing it than :

A=np.array([M]*N) ?

Same question with torch tensor !
Cause, Now, if M is a Variable(torch.tensor), i have to do:

A=torch.autograd.Variable(torch.tensor(np.array([M]*N)))

which is ugly !

Output:
Note, that you need to decide whether you would like to allocate new memory for your expanded array or whether you simply require a new view of the existing memory of the original array.

In PyTorch, this distinction gives rise to the two methods expand() and repeat(). The former only creates a new view on the existing tensor where a dimension of size one is expanded to a larger size by setting the stride to 0. Any dimension of size 1 can be expanded to an arbitrary value without allocating new memory. In contrast, the latter copies the original data and allocates new memory.

In PyTorch, you can use expand() and repeat() as follows for your purposes:

import torch

L = 10
N = 20
A = torch.randn(L,L)
A.expand(N, L, L) # specifies new size
A.repeat(N,1,1) # specifies number of copies

In Numpy, there are a multitude of ways to achieve what you did above in a more elegant and efficient manner. For your particular purpose, I would recommend np.tile() over np.repeat(), since np.repeat() is designed to operate on the particular elements of an array, while np.tile() is designed to operate on the entire array. Hence,

import numpy as np

L = 10
N = 20
A = np.random.rand(L,L)
np.tile(A,(N, 1, 1))


Instruction: How to parallelize RNN function in Pytorch with DataParallel

Input:
Here's an RNN model to run character based language generation:

class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size, n_layers):
        super(RNN, self).__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.output_size = output_size
        self.n_layers = n_layers

        self.encoder = nn.Embedding(input_size, hidden_size)
        self.GRU = nn.GRU(hidden_size, hidden_size, n_layers, batch_first=True)
        self.decoder = nn.Linear(hidden_size, output_size)

    def forward(self, input, batch_size):
        self.init_hidden(batch_size)
        input = self.encoder(input)
        output, self.hidden = self.GRU(input, self.hidden)
        output = self.decoder(output.view(batch_size, self.hidden_size))
        return output

    def init_hidden(self, batch_size):
        self.hidden = Variable(torch.randn(self.n_layers, batch_size, self.hidden_size).cuda())

I instantiate the model using DataParallel, to split the batch of inputs across my 4 GPUs:

net = torch.nn.DataParallel(RNN(n_chars, hidden_size, n_chars, n_layers)).cuda()

Here's the full code.

Unfortunately, DataParallel requires the inputs to have batch_size as the first dimension, but GRU function expects hidden tensor to have batch_size as second dimension:

output, self.hidden = self.GRU(input, self.hidden)

The code as is throws the following error (note the printouts showing that encoder is correctly executed on 4 GPUs):

...
forward function: encoding input of shape: (16L, 1L)
forward function: encoding input of shape: (16L, 1L)
forward function: encoding input of shape: (16L,
forward function: encoding input of shape:

forward function: GRU processing input of shape:
1L)
 ( (16L, 16L1L, 1L), 100L)
forward function: GRU processing input of shape:
 (16L, 1L,
forward function: GRU processing input of shape:100L)
 (16L
forward function: GRU processing input of shape:, 1L, 100L) (
16L, 1L, 100L)

Traceback (most recent call last):
  File "gru2.py", line 166, in <module>
    output = net(c, batch_size)
  File "/root/miniconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 206, in __call__
    result = self.forward(*input, **kwargs)
  File "/root/miniconda2/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py", line 61, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/root/miniconda2/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py", line 71, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs)
  File "/root/miniconda2/lib/python2.7/site-packages/torch/nn/parallel/parallel_apply.py", line 45, in parallel_apply
    raise output
RuntimeError: Expected hidden size (2, 16L, 100), got (2L, 64L, 100L)

Here the model has 2 layers, batch_size=64, and hidden_size = 100.

How do I parallelize the GRU operation in the forward function?

Output:
You can simply set the parameter dim=1, e.g.

net = torch.nn.DataParallel(RNN(n_chars, hidden_size, n_chars, n_layers), dim=1).cuda()


Instruction: How to convert a list of strings into a tensor in pytorch?

Input:
I am working on classification problem in which I have a list of strings as class labels and I want to convert them into a tensor. So far I have tried converting the list of strings into a numpy array using the np.array function provided by the numpy module.

truth = torch.from_numpy(np.array(truths))

but I am getting the following error.

RuntimeError: can't convert a given np.ndarray to a tensor - it has an invalid type. The only supported types are: double, float, int64, int32, and uint8.

Can anybody suggest an alternative approach? Thanks

Output:
Unfortunately, you can't right now. And I don't think it is a good idea since it will make PyTorch clumsy. A popular workaround could convert it into numeric types using sklearn.
Here is a short example:
from sklearn import preprocessing
import torch

labels = ['cat', 'dog', 'mouse', 'elephant', 'pandas']
le = preprocessing.LabelEncoder()
targets = le.fit_transform(labels)
# targets: array([0, 1, 2, 3, 4])

targets = torch.as_tensor(targets)
# targets: tensor([0, 1, 2, 3, 4])

Since you may need the conversion between true labels and transformed labels, it is good to store the variable le.


Instruction: How to handle the BatchNorm layer when training fully convolutional networks by finetuning?

Input:
Training fully convolutional nerworks (FCNs) for pixelwise semantic segmentation is very memory intensive. So we often use batchsize=1 for traing FCNs. However, when we finetune the pretrained networks with BatchNorm (BN) layers, batchsize=1 doesn't make sense for the BN layers. So, how to handle the BN layers?

Some options:

delete the BN layers (merge the BN layers with the preceding layers for the pretrained model)
Freeze the parameters and statistics of the BN layers
....

which is better and any demo for implementation in pytorch/tf/caffe?

Output:
Having only one element will make the batch normalization zero if epsilon is non-zero (variance is zero, mean will be same as input).
Its better to delete the BN layers from the network and try the activation function SELU (scaled exponential linear units). This is from the paper 'Self normalizing neural networks' (SNNs).

Quote from the paper:

  While batch normalization requires explicit normalization, neuron
  activations of SNNs automatically  converge  towards  zero  mean  and
  unit  variance.   The  activation function of SNNs are “scaled
  exponential linear units” (SELUs), which induce self-normalizing
  properties.

The SELU is defined as:

def selu(x, name="selu"):
  alpha = 1.6732632423543772848170429916717
  scale = 1.0507009873554804934193349852946
  return scale * tf.where(x >= 0.0, x, alpha * tf.nn.elu(x))