Instruction: How to sample similar vectors given a vector and cosine similarity in pytorch?

Input:
I have this vector
>>> vec
tensor([[0.2677, 0.1158, 0.5954, 0.9210, 0.3622, 0.4081, 0.4477, 0.7930, 0.1161,
         0.5111, 0.2010, 0.3680, 0.1162, 0.1563, 0.4478, 0.9732, 0.7962, 0.0873,
         0.9793, 0.9382, 0.9468, 0.0851, 0.7601, 0.0322, 0.7553, 0.4025, 0.3627,
         0.5706, 0.3015, 0.1344, 0.8343, 0.8187, 0.4287, 0.5785, 0.9527, 0.1632,
         0.2890, 0.5411, 0.5319, 0.7163, 0.3166, 0.5717, 0.5018, 0.5368, 0.3321]])

using this vector I want to generate 15 vectors which have a cosine similarity greater than 80%.
How can I do this in pytorch?

Output:
I modified the answer here, adding an extra dimension and converting from numpy to torch.
def torch_cos_sim(v,cos_theta,n_vectors = 1,EXACT = True):

   """
   EXACT - if True, all vectors will have exactly cos_theta similarity.
           if False, all vectors will have >= cos_theta similarity
   v - original vector (1D tensor)
   cos_theta -cos similarity in range [-1,1]
   """

   # unit vector in direction of v
   u = v / torch.norm(v)
   u = u.unsqueeze(0).repeat(n_vectors,1)

   # random vector with elements in range [-1,1]
   r = torch.rand([n_vectors,len(v)])*2 -1

   # unit vector perpendicular to v and u
   uperp = torch.stack([r[i] - (torch.dot(r[i],u[i]) * u[i]) for i in range(len(u))])
   uperp = uperp/ (torch.norm(uperp,dim = 1).unsqueeze(1).repeat(1,v.shape[0]))

   if not EXACT:
    cos_theta = torch.rand(n_vectors)* (1-cos_theta) + cos_theta
    cos_theta = cos_theta.unsqueeze(1).repeat(1,v.shape[0])

   # w is the linear combination of u and uperp with coefficients costheta
   # and sin(theta) = sqrt(1 - costheta**2), respectively:
   w = cos_theta*u + torch.sqrt(1 - torch.tensor(cos_theta)**2)*uperp

   return w

You can check the output with:
vec = torch.rand(54)
output = torch_cos_sim(vec,0.6,n_vectors = 15, EXACT = False)

# test cos similarity
for item in output:
    print(torch.dot(vec,item)/(torch.norm(vec)*torch.norm(item)))


Instruction: Convert keras model architecture to Pytorch

Input:
I am trying to convert a keras model to pytorch for human activity recognition. The keras model could achieve up to 98% accuracy, while Pytorch model could achieve only ~60% accuracy. I couldn’t figure out the problem, first I was thinking of the padding=‘same’ of keras, but I have already adjusted the padding of pytorch already. Can you check what is wrong?
The keras code is as above
model = keras.Sequential()
model.add(layers.Input(shape=[100,12]))
model.add(layers.Conv1D(filters=32, kernel_size=3, padding="same"))
model.add(layers.BatchNormalization())
model.add(layers.ReLU())
model.add(layers.Conv1D(filters=64, kernel_size=3, padding="same"))
model.add(layers.BatchNormalization())
model.add(layers.ReLU())
model.add(layers.MaxPool1D(2))
model.add(layers.LSTM(64))
model.add(layers.Dense(units=128, activation='relu'))
model.add(layers.Dense(13, activation='softmax'))
model.summary()

and my pytorch model code is as below
class CNN(nn.Module):
    def __init__(self):
        super(CNN, self).__init__()
        seq_len = 100
        # output: [, 32, 100]
        self.conv1 = nn.Conv1d(seq_len, 32, kernel_size=3, stride=1, padding=1)
        self.bn1 = nn.BatchNorm1d(32)
        # output: [, 64, 100]
        self.conv2 = nn.Conv1d(32, 64, kernel_size=3, padding=1)
        self.bn2 = nn.BatchNorm1d(64)
        # output: [, 64, 50]
        self.mp = nn.MaxPool1d(kernel_size=2, stride=2)
        # output: [, 64]
        self.lstm = nn.LSTM(6, 64, 1)
        # output: [, 128]
        self.fc1 = nn.Linear(64, 128)
        # output: [, 13]
        self.fc2 = nn.Linear(128, 13)

    def forward(self, x):
        x = self.conv1(x)
        x = self.bn1(x)
        x = F.relu(x)

        x = self.conv2(x)
        x = self.bn2(x)
        x = F.relu(x)
        x = self.mp(x)

        out, _ = self.lstm(x)
        x = out[:, -1, :]

        x = self.fc1(x)
        x = F.relu(x)
        x = self.fc2(x)

        return x

Output:
Since it was too lengthy to write in the comments,I am writing it in the answer, After testing your PyTorch architecture for shape with random tensor of size which you have mentioned: torch.randn(1, 100, 12)(NCH format).
This is the outcome:
input= torch.Size([1, 100, 12])
1st Conv= torch.Size([1, 32, 12])
1st batchNorm= torch.Size([1, 32, 12])
1st relu= torch.Size([1, 32, 12])
2nd Conv= torch.Size([1, 64, 12])
2nd batchnorm= torch.Size([1, 64, 12])
2nd relu= torch.Size([1, 64, 12])
1st maxPool= torch.Size([1, 64, 6])
LSTM= torch.Size([1, 64])
1st FC= torch.Size([1, 128])
3rd relu= torch.Size([1, 128])
2nd FC= torch.Size([1, 13])

It is the 100 channels your network is receiving, and as you mentioned in comments after 1st convolution in Keras it's [batch_size, 100, 32]  but in torch it is changing to [batch_size,32,12]
Change to this:
def __init__(self):
        super(CNN, self).__init__()
        in_channels = 12
        # output: [, 32, 100]
        self.conv1 = nn.Conv1d(in_channels, 32, kernel_size=3, stride=1, padding=1)
        self.bn1 = nn.BatchNorm1d(32)
        # output: [, 64, 100]
        self.conv2 = nn.Conv1d(32, 64, kernel_size=3, padding=1)
        self.bn2 = nn.BatchNorm1d(64)
        # output: [, 64, 50]
        self.mp = nn.MaxPool1d(kernel_size=2, stride=2)
        # output: [, 64]
        self.lstm = nn.LSTM(50, 64, 1)
        # output: [, 128]
        self.fc1 = nn.Linear(64, 128)
        # output: [, 13]
        self.fc2 = nn.Linear(128, 13)
        self.softmax = nn.Softmax()

The output for this will be:
input= torch.Size([1, 12, 100])
1st Conv= torch.Size([1, 32, 100])
1st batchNorm= torch.Size([1, 32, 100])
1st relu= torch.Size([1, 32, 100])
2nd Conv= torch.Size([1, 64, 100])
2nd batchnorm= torch.Size([1, 64, 100])
2nd relu= torch.Size([1, 64, 100])
1st maxPool= torch.Size([1, 64, 50])
LSTM= torch.Size([1, 64])
1st FC= torch.Size([1, 128])
3rd relu= torch.Size([1, 128])
2nd FC= torch.Size([1, 13])


Instruction: Can't do lazy loading with allennlp

Input:
Currently I'm trying to implement lazy loading with allennlp, but can't.
My code is as the followings.
def biencoder_training():
    params = BiEncoderExperiemntParams()
    config = params.opts
    reader = SmallJaWikiReader(config=config)

    # Loading Datasets
    train, dev, test = reader.read('train'), reader.read('dev'), reader.read('test')
    vocab = build_vocab(train)
    vocab.extend_from_instances(dev)

    # TODO: avoid memory consumption and lazy loading
    train, dev, test = list(reader.read('train')), list(reader.read('dev')), list(reader.read('test'))

    train_loader, dev_loader, test_loader = build_data_loaders(config, train, dev, test)
    train_loader.index_with(vocab)
    dev_loader.index_with(vocab)

    embedder = emb_returner()
    mention_encoder, entity_encoder = Pooler_for_mention(word_embedder=embedder), \
                                      Pooler_for_cano_and_def(word_embedder=embedder)

    model = Biencoder(mention_encoder, entity_encoder, vocab)

    trainer = build_trainer(lr=config.lr,
                            num_epochs=config.num_epochs,
                            model=model,
                            train_loader=train_loader,
                            dev_loader=dev_loader)
    trainer.train()

    return model

When I commented-out  train, dev, test = list(reader.read('train')), list(reader.read('dev')), list(reader.read('test')), iterator doesn't work and training is conducted with 0 sample.
Building the vocabulary
100it [00:00, 442.15it/s]01, 133.57it/s]
building vocab: 100it [00:01, 95.84it/s]
100it [00:00, 413.40it/s]
100it [00:00, 138.38it/s]
You provided a validation dataset but patience was set to None, meaning that early stopping is disabled
0it [00:00, ?it/s]
0it [00:00, ?it/s]

I'd like to know if there is any solution for avoid this.
Thanks.

Supplement, added at fifth, May.
Currently I am trying to avoid putting all of each sample data on top of memory before training the model.
So I have implemented the _read method as a generator. My understanding is that by calling this method and wrapping it with SimpleDataLoader, I can actually pass the data to the model.
In the DatasetReader, the code for the _read method looks like this. It is my understanding that this is intended to be a generator that avoids memory consumption.
    @overrides
    def _read(self, train_dev_test_flag: str) -> Iterator[Instance]:
        '''
        :param train_dev_test_flag: 'train', 'dev', 'test'
        :return: list of instances
        '''
        if train_dev_test_flag == 'train':
            dataset = self._train_loader()
            random.shuffle(dataset)
        elif train_dev_test_flag == 'dev':
            dataset = self._dev_loader()
        elif train_dev_test_flag == 'test':
            dataset = self._test_loader()
        else:
            raise NotImplementedError(
                "{} is not a valid flag. Choose from train, dev and test".format(train_dev_test_flag))

        if self.config.debug:
            dataset = dataset[:self.config.debug_data_num]

        for data in tqdm(enumerate(dataset)):
            data = self._one_line_parser(data=data, train_dev_test_flag=train_dev_test_flag)
            yield self.text_to_instance(data)

Also, build_data_loaders actually looks like this.
def build_data_loaders(config,
    train_data: List[Instance],
    dev_data: List[Instance],
    test_data: List[Instance]) -> Tuple[DataLoader, DataLoader, DataLoader]:

    train_loader = SimpleDataLoader(train_data, config.batch_size_for_train, shuffle=False)
    dev_loader = SimpleDataLoader(dev_data, config.batch_size_for_eval, shuffle=False)
    test_loader = SimpleDataLoader(test_data, config.batch_size_for_eval, shuffle=False)

    return train_loader, dev_loader, test_loader

But, by somewhat reason I don't know, this code doesn't work.
def biencoder_training():
    params = BiEncoderExperiemntParams()
    config = params.opts
    reader = SmallJaWikiReader(config=config)

    # Loading Datasets
    train, dev, test = reader.read('train'), reader.read('dev'), reader.read('test')
    vocab = build_vocab(train)
    vocab.extend_from_instances(dev)

    train_loader, dev_loader, test_loader = build_data_loaders(config, train, dev, test)
    train_loader.index_with(vocab)
    dev_loader.index_with(vocab)

    embedder = emb_returner()
    mention_encoder, entity_encoder = Pooler_for_mention(word_embedder=embedder), \
                                      Pooler_for_cano_and_def(word_embedder=embedder)

    model = Biencoder(mention_encoder, entity_encoder, vocab)

    trainer = build_trainer(lr=config.lr,
                            num_epochs=config.num_epochs,
                            model=model,
                            train_loader=train_loader,
                            dev_loader=dev_loader)
    trainer.train()

    return model

In this code, the SimpleDataLoader will wrap the generator type as it is. I would like to do the lazy loading that allennlp did in the 0.9 version.
But this code iterates training over 0 instances, so currently I have added
train, dev, test = list(reader.read('train')), list(reader.read('dev')), list(reader.read('test'))
before
train_loader, dev_loader, test_loader = build_data_loaders(config, train, dev, test).
And it works. But this means that I can't train or evaluate the model until I have all the instances in memory. Rather, I want each batch to be called into memory only when it is time to train.

Output: The SimpleDataLoader is not capable of lazy loading. You should use the MultiProcessDataLoader instead. Setting max_instances_in_memory to a non-zero integer (usually some multiple of your batch size) will trigger lazy loading.


Instruction: Why is this tensorflow training taking so long?

Input:
I'm learning DRL with the book Deep Reinforcement Learning in Action. In chapter 3, they present the simple game Gridworld (instructions here, in the rules section) with the corresponding code in PyTorch.
I've experimented with the code and it takes less than 3 minutes to train the network with 89% of wins (won 89 of 100 games after training).

As an exercise, I have migrated the code to tensorflow. All the code is here.
The problem is that with my tensorflow port it takes near 2 hours to train the network with a win rate of 84%. Both versions are using the only CPU to train (I don't have GPU)

Training loss figures seem correct and also the rate of a win (we have to take into consideration that the game is random and can have impossible states). The problem is the performance of the overall process.
I'm doing something terribly wrong, but what?
The main differences are in the training loop, in torch is this:
        loss_fn = torch.nn.MSELoss()
        learning_rate = 1e-3
        optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
        ....
        Q1 = model(state1_batch)
        with torch.no_grad():
            Q2 = model2(state2_batch) #B

        Y = reward_batch + gamma * ((1-done_batch) * torch.max(Q2,dim=1)[0])
        X = Q1.gather(dim=1,index=action_batch.long().unsqueeze(dim=1)).squeeze()
        loss = loss_fn(X, Y.detach())
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

and in the tensorflow version:
        loss_fn = tf.keras.losses.MSE
        learning_rate = 1e-3
        optimizer = tf.keras.optimizers.Adam(learning_rate)
        ...
        Q2 = model2(state2_batch) #B
        with tf.GradientTape() as tape:
            Q1 = model(state1_batch)
            Y = reward_batch + gamma * ((1-done_batch) * tf.math.reduce_max(Q2, axis=1))
            X = [Q1[i][action_batch[i]] for i in range(len(action_batch))]
            loss = loss_fn(X, Y)
        grads = tape.gradient(loss, model.trainable_variables)
        optimizer.apply_gradients(zip(grads, model.trainable_variables))

Why is the training taking so long?

Output:
Why is TensorFlow slow
TensorFlow has 2 execution modes: eager execution, and graph mode. TensorFlow default behavior, since version 2, is to default to eager execution. Eager execution is great as it enables you to write code close to how you would write standard python. It's easier to write, and it's easier to debug. Unfortunately, it's really not as fast as graph mode.
So the idea is, once the function is prototyped in eager mode, to make TensorFlow execute it in graph mode. For that you can use tf.function. tf.function compiles a callable into a TensorFlow graph. Once the function is compiled into a graph, the performance gain is usually quite important. The recommended approach when developing in TensorFlow is the following:

Debug in eager mode, then decorate with @tf.function.
Don't rely on Python side effects like object mutation or list appends.
tf.function works best with TensorFlow ops; NumPy and Python calls are converted to constants.

I would add: think about the critical parts of your program, and which ones should be converted first into graph mode. It's usually the parts where you call a model to get a result. It's where you will see the best improvements.
You can find more information in the following guides:

Better performance with tf.function
Introduction to graphs and tf.function

Applying tf.function to your code
So, there are at least two things you can change in your code to make it run quite faster:

The first one is to not use model.predict on a small amount of data. The function is made to work on a huge dataset or on a generator. (See this comment on Github). Instead, you should call the model directly, and for performance enhancement, you can wrap the call to the model in a tf.function.

Model.predict is a top-level API designed for batch-predicting outside of any loops, with the fully-features of the Keras APIs.

The second one is to make your training step a separate function, and to decorate that function with @tf.function.

So, I would declare the following things before your training loop:
# to call instead of model.predict
model_func = tf.function(model)

def get_train_func(model, model2, loss_fn, optimizer):
    """Wrapper that creates a train step using the two model passed"""
    @tf.function
    def train_func(state1_batch, state2_batch, done_batch, reward_batch, action_batch):
        Q2 = model2(state2_batch) #B
        with tf.GradientTape() as tape:
            Q1 = model(state1_batch)
            Y = reward_batch + gamma * ((1-done_batch) * tf.math.reduce_max(Q2, axis=1))
            # gather is more efficient than a list comprehension, and needed in a tf.function
            X = tf.gather(Q1, action_batch, batch_dims=1)
            loss = loss_fn(X, Y)
        grads = tape.gradient(loss, model.trainable_variables)
        optimizer.apply_gradients(zip(grads, model.trainable_variables))
        return loss
    return train_func

# train step is a callable
train_step = get_train_func(model, model2, loss_fn, optimizer)

And you can use that function in your training loop:
if len(replay) > batch_size:
    minibatch = random.sample(replay, batch_size)
    state1_batch = np.array([s1 for (s1,a,r,s2,d) in minibatch]).reshape((batch_size, 64))
    action_batch = np.array([a for (s1,a,r,s2,d) in minibatch])   #TODO: Posibles diferencies
    reward_batch = np.float32([r for (s1,a,r,s2,d) in minibatch])
    state2_batch = np.array([s2 for (s1,a,r,s2,d) in minibatch]).reshape((batch_size, 64))
    done_batch = np.array([d for (s1,a,r,s2,d) in minibatch]).astype(np.float32)

    loss = train_step(state1_batch, state2_batch, done_batch, reward_batch, action_batch)
    losses.append(loss)

There are other changes that you could make to make your code more TensorFlowesque, but with those modifications, your code takes ~2 minutes on my CPU. (with a 97% win rate).


Instruction: Sequential network with the VGG layers

Input:
I want to have a sequential network with the characteristics of VGG network (I want to pass my network to another function, which doesn't support VGG objects and supports nn.sequential).
I added the function getSequentialVersion method to VGG class to have the sequential network with the linear layer. However, apparently, there is a size mismatch in the network.
'''VGG for CIFAR10. FC layers are removed.
(c) YANG, Wei
'''
import torch.nn as nn
import torch.utils.model_zoo as model_zoo
import math

__all__ = [
    'VGG','vgg16_bn',
]

model_urls = {
    'vgg16': 'https://download.pytorch.org/models/vgg16-397923af.pth',
}

class VGG(nn.Module):
    def __init__(self, features, num_classes=1000, cfg_type=None, batch_norm=False, **kwargs):
        super(VGG, self).__init__()
        self.features = features
        self.classifier = nn.Linear(512, num_classes)
        self._initialize_weights()
        self.cfg_type = cfg_type
        self.batch_norm = batch_norm

    def forward(self, x):
        x = self.features(x)
        x = x.view(x.size(0), -1)
        x = self.classifier(x)
        return x

    def _initialize_weights(self):
        for m in self.modules():
            if isinstance(m, nn.Conv2d):
                n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels
                m.weight.data.normal_(0, math.sqrt(2. / n))
                if m.bias is not None:
                    m.bias.data.zero_()
            elif isinstance(m, nn.BatchNorm2d):
                m.weight.data.fill_(1)
                m.bias.data.zero_()
            elif isinstance(m, nn.Linear):
                n = m.weight.size(1)
                m.weight.data.normal_(0, 0.01)
                m.bias.data.zero_()

    def getSequentialVersion(self):
        return make_layers(cfg[self.cfg_type], batch_norm=self.batch_norm, flag=True)

def make_layers(cfg, batch_norm=False, flag=False):
    layers = []
    in_channels = 3
    for v in cfg:
        if v == 'M':
            layers += [nn.MaxPool2d(kernel_size=2, stride=2)]
        else:
            conv2d = nn.Conv2d(in_channels, v, kernel_size=3, padding=1, bias=False)
            if batch_norm:
                layers += [conv2d, nn.BatchNorm2d(v), nn.ReLU(inplace=True)]
            else:
                layers += [conv2d, nn.ReLU(inplace=True)]
            in_channels = v
    if flag:
        #for Cifar10
        layers += [nn.Linear(512, 10)]
    return nn.Sequential(*layers)

cfg = {
    'A': [64, 'M', 128, 'M', 256, 256, 'M', 512, 512, 'M', 512, 512, 'M'],
    'B': [64, 64, 'M', 128, 128, 'M', 256, 256, 'M', 512, 512, 'M', 512, 512, 'M'],
    'D': [64, 64, 'M', 128, 128, 'M', 256, 256, 256, 'M', 512, 512, 512, 'M', 512, 512, 512, 'M'],
    'E': [64, 64, 'M', 128, 128, 'M', 256, 256, 256, 256, 'M', 512, 512, 512, 512, 'M', 512, 512, 512, 512, 'M'],
}

def vgg16_bn(**kwargs):
    """VGG 16-layer model (configuration "D") with batch normalization"""
    print("VGG16-bn")
    model = VGG(make_layers(cfg['D'], batch_norm=True), cfg_type='D', batch_norm=True,**kwargs)
    return model

When I call summary(net, ( 3, 32, 32)) (for cifar10), I get the mismatch error. In other words, the main problem is when I add this line layers+= [nn.linear(512, 10)].
Can anyone help me? Thanks a lot.
The error message:
  File "./main.py", line 284, in <module>
    summary(net, ( 3, 32, 32))
  File "./anaconda3/envs/my_env/lib/python3.8/site-packages/torchsummary/torchsummary.py", line 72, in summary
    model(*x)
  File ".anaconda3/envs/my_env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "./anaconda3/envs/my_env/lib/python3.8/site-packages/torch/nn/modules/container.py", line 119, in forward
    input = module(input)
  File "./anaconda3/envs/my_env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "./anaconda3/envs/my_env/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 94, in forward
    return F.linear(input, self.weight, self.bias)
  File "./envs/my_env/lib/python3.8/site-packages/torch/nn/functional.py", line 1753, in linear
    return torch._C._nn.linear(input, weight, bias)
RuntimeError: mat1 dim 1 must match mat2 dim 0

Additional information:
This is how exactly I initialize and use my network:
net = vgg16_bn(depth=args.depth,
                  num_classes=num_classes,
                  growthRate=args.growthRate,
                  compressionRate=args.compressionRate,
                  widen_factor=args.widen_factor,
                  dropRate=args.dropRate,
                  base_width=args.base_width,
                  cardinality=args.cardinality).getSequentialVersion()
net = net.to(args.device)
module_names = ''
if hasattr(net, 'features'):
    module_names = 'features'
elif hasattr(net, 'children'):
    module_names = 'children'
else:
    print('unknown net modules...')

summary(net, ( 3, 32, 32))

Output:
The problem is quite simple. When flag=True (as in getSequentialVersion()), there's a missing Flatten operation. Therefore, to fix the problem, you need to add this operation like this:
if flag:
    # for Cifar10
    layers += [nn.Flatten(), nn.Linear(512, 10)]  # <<< add Flatten before Linear

In the forward call, you can see the flatten in its view form:
def forward(self, x):
    x = self.features(x)
    x = x.view(x.size(0), -1)  # here, equivalent to torch.flatten(x, 1)
    x = self.classifier(x)
    return x

and this is what was missing when you were transforming the layers to Sequential.


Instruction: Pytorch custom CUDA extension build fails for torch 1.6.0 or higher

Input:
I have a custom CUDA extension for pytorch (https://pytorch.org/tutorials/advanced/cpp_extension.html), which used to work fine with pytorch1.4, CUDA10.1, and Titan Xp GPUs. However, recently we changed our system to new A40 GPUs and CUDA11.1. When I try to build my custom pytorch extension using CUDA11.1, pytorch 1.8.1, gcc 9.3.0, and Ubuntu 20.04 I get the following errors:
$ python3 setup.py install
running install
running bdist_egg
running egg_info
creating cuda_test.egg-info
writing cuda_test.egg-info/PKG-INFO
writing dependency_links to cuda_test.egg-info/dependency_links.txt
writing top-level names to cuda_test.egg-info/top_level.txt
writing manifest file 'cuda_test.egg-info/SOURCES.txt'
reading manifest file 'cuda_test.egg-info/SOURCES.txt'
writing manifest file 'cuda_test.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_ext
building 'cuda_test' extension
creating /path/to/code/cuda/test/build
creating /path/to/code/cuda/test/build/temp.linux-x86_64-3.7
Emitting ninja build file /path/to/code/cuda/test/build/temp.linux-x86_64-3.7/build.ninja...
Compiling objects...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/1] /cm/shared/apps/cuda11.1/toolkit/11.1.1/bin/nvcc --generate-dependencies-with-compile --dependency-output /path/to/code/cuda/test/build/temp.linux-x86_64-3.7/test_cuda.o.d -I/path/to/code/venv/lib/python3.7/site-packages/torch/include -I/path/to/code/venv/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/path/to/code/venv/lib/python3.7/site-packages/torch/include/TH -I/path/to/code/venv/lib/python3.7/site-packages/torch/include/THC -I/cm/shared/apps/cuda11.1/toolkit/11.1.1/include -I/path/to/code/venv/include/python3.7m -c -c /path/to/code/cuda/test/test_cuda.cu -o /path/to/code/cuda/test/build/temp.linux-x86_64-3.7/test_cuda.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=cuda_test -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++14
FAILED: /path/to/code/cuda/test/build/temp.linux-x86_64-3.7/test_cuda.o
/cm/shared/apps/cuda11.1/toolkit/11.1.1/bin/nvcc --generate-dependencies-with-compile --dependency-output /path/to/code/cuda/test/build/temp.linux-x86_64-3.7/test_cuda.o.d -I/path/to/code/venv/lib/python3.7/site-packages/torch/include -I/path/to/code/venv/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/path/to/code/venv/lib/python3.7/site-packages/torch/include/TH -I/path/to/code/venv/lib/python3.7/site-packages/torch/include/THC -I/cm/shared/apps/cuda11.1/toolkit/11.1.1/include -I/path/to/code/venv/include/python3.7m -c -c /path/to/code/cuda/test/test_cuda.cu -o /path/to/code/cuda/test/build/temp.linux-x86_64-3.7/test_cuda.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=cuda_test -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++14
/cm/shared/apps/cuda11.1/toolkit/11.1.1/include/thrust/detail/complex/arithmetic.h(256): error: identifier "FLT_MIN" is undefined

/cm/shared/apps/cuda11.1/toolkit/11.1.1/include/thrust/detail/complex/arithmetic.h(274): error: identifier "DBL_MIN" is undefined

/cm/shared/apps/cuda11.1/toolkit/11.1.1/include/thrust/detail/complex/catrig.h(190): error: identifier "DBL_EPSILON" is undefined

/cm/shared/apps/cuda11.1/toolkit/11.1.1/include/thrust/detail/complex/catrig.h(228): error: identifier "DBL_EPSILON" is undefined

/cm/shared/apps/cuda11.1/toolkit/11.1.1/include/thrust/detail/complex/catrig.h(243): error: identifier "DBL_EPSILON" is undefined

/cm/shared/apps/cuda11.1/toolkit/11.1.1/include/thrust/detail/complex/catrig.h(293): error: identifier "DBL_EPSILON" is undefined

/cm/shared/apps/cuda11.1/toolkit/11.1.1/include/thrust/detail/complex/catrig.h(406): error: identifier "DBL_EPSILON" is undefined

/cm/shared/apps/cuda11.1/toolkit/11.1.1/include/thrust/detail/complex/catrig.h(498): error: identifier "DBL_MAX" is undefined

/cm/shared/apps/cuda11.1/toolkit/11.1.1/include/thrust/detail/complex/catrig.h(562): error: identifier "DBL_MAX_EXP" is undefined

/cm/shared/apps/cuda11.1/toolkit/11.1.1/include/thrust/detail/complex/catrig.h(565): error: identifier "DBL_MANT_DIG" is undefined

/cm/shared/apps/cuda11.1/toolkit/11.1.1/include/thrust/detail/complex/catrig.h(630): error: identifier "DBL_EPSILON" is undefined

/cm/shared/apps/cuda11.1/toolkit/11.1.1/include/thrust/detail/complex/catrigf.h(119): error: identifier "FLT_EPSILON" is undefined

/cm/shared/apps/cuda11.1/toolkit/11.1.1/include/thrust/detail/complex/catrigf.h(137): error: identifier "FLT_EPSILON" is undefined

/cm/shared/apps/cuda11.1/toolkit/11.1.1/include/thrust/detail/complex/catrigf.h(147): error: identifier "FLT_EPSILON" is undefined

/cm/shared/apps/cuda11.1/toolkit/11.1.1/include/thrust/detail/complex/catrigf.h(170): error: identifier "FLT_EPSILON" is undefined

/cm/shared/apps/cuda11.1/toolkit/11.1.1/include/thrust/detail/complex/catrigf.h(249): error: identifier "FLT_EPSILON" is undefined

/cm/shared/apps/cuda11.1/toolkit/11.1.1/include/thrust/detail/complex/catrigf.h(327): error: identifier "FLT_MAX" is undefined

/cm/shared/apps/cuda11.1/toolkit/11.1.1/include/thrust/detail/complex/catrigf.h(375): error: identifier "FLT_MAX_EXP" is undefined

/cm/shared/apps/cuda11.1/toolkit/11.1.1/include/thrust/detail/complex/catrigf.h(377): error: identifier "FLT_MANT_DIG" is undefined

/cm/shared/apps/cuda11.1/toolkit/11.1.1/include/thrust/detail/complex/catrigf.h(420): error: identifier "FLT_EPSILON" is undefined

I also wrote a simple test code to verify that my larger CPP/CUDA code isn't the culprit, which produced the same error messages. I also checked if arithmetic.h and catrig.h include <cfloat>, which should provide the {FLT,DBL}_{MIN,MAX,EPSILON,MANT_DIG} definitions but this looks all normal, since it's standard NVIDIA code.
Let me know if anyone has encountered a similar problem or would know a solution.
---- UPDATE ----
Here are a couple of more things that I've tried:

The CUDA code compiles when I use CUDA10.1, pytorch 1.4.0, gcc 9.3.0, and Ubuntu 20.04.
Using pytorch 1.5.1 instead generates the following error:
/usr/include/c++/9/bits/stl_function.h(437): error: identifier "__builtin_is_constant_evaluated" is undefined
but this can be solved by downgrading gcc to version 7.5.
Using pytorch 1.6.0 or higher instead always results in the errors reported in the beginning, even when using gcc-7.

Output: I found the issue. The Intel MKL module wasn't loaded properly and caused the error. After fixing this the compilation worked just fine also with CUDA 11.1 and pytorch 1.8.1!


Instruction: numpy equivalent code of unsqueeze and expand from torch tensor method

Input:
I have these 2 tensors
box_a = torch.randn(1,4)
box_b = torch.randn(1,4)

and i have a code in pytorch
box_a[:, 2:].unsqueeze(1).expand(1, 1, 2)

but i want to convert the above code in numpy
for box_a and box_b i can do something like this
  box_a = numpy.random.randn(1,4)
  box_b = numpy.random.randn(1,4)

But what about this
box_a[:, 2:].unsqueeze(1).expand(1, 1, 2)

Output:
solved it
box_a = np.random.randn(1,4)
box_b = np.random.randn(1,4)
max_xy = np.broadcast_to(np.expand_dims(box_a[:, 2:],axis=1),(1,1,2))


Instruction: How do I one hot encode along a specific dimension using PyTorch?

Input:
I have a tensor of size [3, 15, 136], where:

3 is batch size
15 - sequence length and
136 is tokens

I want to one-hot my tensor using the probabilities in the tokens dimension (136). To do so I want to extract the tokens dimension for each letter in sequence length and put 1 to the largest possibility and mark all other tokens as 0.

Output:
You can use PyTorch's one_hot function to achieve this:
import torch.nn.functional as F

t = torch.rand(3, 15, 136)

F.one_hot(t.argmax(dim=2), 136)


Instruction: Attention weighted aggregation

Input:
Let the tensor shown below be the representation of two sentences (batch_size = 2) composed with 3 words (max_lenght = 3) and each word being represented by vectors of dimension equal to 5 (hidden_size = 5) obtained as output from a neural network:
net_output
# tensor([[[0.7718, 0.3856, 0.2545, 0.7502, 0.5844],
#          [0.4400, 0.3753, 0.4840, 0.2483, 0.4751],
#          [0.4927, 0.7380, 0.1502, 0.5222, 0.0093]],

#         [[0.5859, 0.0010, 0.2261, 0.6318, 0.5636],
#          [0.0996, 0.2178, 0.9003, 0.4708, 0.7501],
#          [0.4244, 0.7947, 0.5711, 0.0720, 0.1106]]])

Also consider the following attention scores:
att_scores
# tensor([[0.2425, 0.5279, 0.2295],
#         [0.2461, 0.4789, 0.2751]])

Which efficient approach allows obtaining the aggregation of vectors in net_output weighted by att_scores resulting in a vector of shape (2, 5)?

Output:
This should work:
weighted = (net_output * att_scores[..., None]).sum(axis = 1)

Uses broadcasting to (elementwise) multiply the attention weights to each vector and aggregates (them by summing) all vectors in a batch.


Instruction: issue with calculating accuracy

Input:
i'm using Torch Metrics to try to calculate the accuracy of my model. But i'm getting this error. I tried using .to(device="cuda:0") but I got a cuda initialization error. I also tried using .cuda() but that didn't work either. I'm using PyTorch lightning with a Titan Xp GPU. Im using a mish activation function with the Movie-lens data set.
code:

# %% [markdown]
# # Data Preprocessing
#
# Before we start building and training our model, let's do some preprocessing to get the data in the required format.

# %% [code] {"_kg_hide-input":true,"_kg_hide-output":true}
import pandas as pd
import numpy as np
from tqdm.notebook import tqdm

import torch
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader
import pytorch_lightning as pl
import torch.nn.functional as F
from pytorch_lightning.callbacks import EarlyStopping
import wandb
import torchmetrics

wandb.init(project="Mocean-Recommendor",entity="maxall4")

config = wandb.config

def mish(x):
    return (x*torch.tanh(F.softplus(x)))

np.random.seed(123)

# %% [markdown]
# First, we import the ratings dataset.

# %% [code]
ratings = pd.read_csv('rating.csv',
                      parse_dates=['timestamp'])

# %% [markdown]
# In order to keep memory usage manageable within Kaggle's kernel, we will only use data from 30% of the users in this dataset. Let's randomly select 30% of the users and only use data from the selected users.

# %% [code]
rand_userIds = np.random.choice(ratings['userId'].unique(),
                                size=int(len(ratings['userId'].unique())*0.3),
                                replace=False)

ratings = ratings.loc[ratings['userId'].isin(rand_userIds)]

print('There are {} rows of data from {} users'.format(len(ratings), len(rand_userIds)))

# %% [code]
ratings.sample(5)

# %% [code]
ratings['rank_latest'] = ratings.groupby(['userId'])['timestamp'] \
                                .rank(method='first', ascending=False)

train_ratings = ratings[ratings['rank_latest'] != 1]
test_ratings = ratings[ratings['rank_latest'] == 1]

# drop columns that we no longer need
train_ratings = train_ratings[['userId', 'movieId', 'rating']]
test_ratings = test_ratings[['userId', 'movieId', 'rating']]

# %% [markdown]
# ### Converting the dataset into an implicit feedback dataset

# %% [code]
train_ratings.loc[:, 'rating'] = 1

train_ratings.sample(5)

# %% [markdown]
# The code below generates 4 negative samples for each row of data. In other words, the ratio of negative to positive samples is 4:1. This ratio is chosen arbitrarily but I found that it works rather well (feel free to find the best ratio yourself!)

# %% [code]
# Get a list of all movie IDs
all_movieIds = ratings['movieId'].unique()

# Placeholders that will hold the training data
users, items, labels = [], [], []

# This is the set of items that each user has interaction with
user_item_set = set(zip(train_ratings['userId'], train_ratings['movieId']))

# 4:1 ratio of negative to positive samples
num_negatives = 4

for (u, i) in tqdm(user_item_set):
    users.append(u)
    items.append(i)
    labels.append(1) # items that the user has interacted with are positive
    for _ in range(num_negatives):
        # randomly select an item
        negative_item = np.random.choice(all_movieIds)
        # check that the user has not interacted with this item
        while (u, negative_item) in user_item_set:
            negative_item = np.random.choice(all_movieIds)
        users.append(u)
        items.append(negative_item)
        labels.append(0) # items not interacted with are negative

# %% [code]
class MovieLensTrainDataset(Dataset):
    """MovieLens PyTorch Dataset for Training

    Args:
        ratings (pd.DataFrame): Dataframe containing the movie ratings
        all_movieIds (list): List containing all movieIds

    """

    def __init__(self, ratings, all_movieIds):
        self.users, self.items, self.labels = self.get_dataset(ratings, all_movieIds)

    def __len__(self):
        return len(self.users)

    def __getitem__(self, idx):
        return self.users[idx], self.items[idx], self.labels[idx]

    def get_dataset(self, ratings, all_movieIds):
        users, items, labels = [], [], []
        user_item_set = set(zip(ratings['userId'], ratings['movieId']))

        num_negatives = 4
        for u, i in user_item_set:
            users.append(u)
            items.append(i)
            labels.append(1)
            for _ in range(num_negatives):
                negative_item = np.random.choice(all_movieIds)
                while (u, negative_item) in user_item_set:
                    negative_item = np.random.choice(all_movieIds)
                users.append(u)
                items.append(negative_item)
                labels.append(0)

        return torch.tensor(users), torch.tensor(items), torch.tensor(labels)

# %% [code]
acc_metric = torchmetrics.Accuracy()
class NCF(pl.LightningModule):
    """ Neural Collaborative Filtering (NCF)

        Args:
            num_users (int): Number of unique users
            num_items (int): Number of unique items
            ratings (pd.DataFrame): Dataframe containing the movie ratings for training
            all_movieIds (list): List containing all movieIds (train + test)
    """

    def __init__(self, num_users, num_items, ratings, all_movieIds):
        super().__init__()
        self.user_embedding = nn.Embedding(num_embeddings=num_users, embedding_dim=8)
        self.item_embedding = nn.Embedding(num_embeddings=num_items, embedding_dim=8)
        self.fc1 = nn.Linear(in_features=16, out_features=64)
        self.fc2 = nn.Linear(in_features=64, out_features=32)
        self.output = nn.Linear(in_features=32, out_features=1)
        self.ratings = ratings
        self.all_movieIds = all_movieIds

    def on_validation_end(self,outputs):
        loss = torch.stack([x['val_loss'] for x in outputs]).mean()
        return { 'loss' : loss }
    def forward(self, user_input, item_input):

        # Pass through embedding layers
        user_embedded = self.user_embedding(user_input)
        item_embedded = self.item_embedding(item_input)

        # Concat the two embedding layers
        vector = torch.cat([user_embedded, item_embedded], dim=-1)

        # Pass through dense layer
        vector = mish(self.fc1(vector))
        vector = mish(self.fc2(vector))

        # Output layer
        pred = nn.Sigmoid()(self.output(vector))

        return pred

    def training_step(self, batch, batch_idx):
        user_input, item_input, labels = batch
        predicted_labels = self(user_input, item_input)
        loss = nn.BCELoss()(predicted_labels, labels.view(-1, 1).float())

        acc = acc_metric(predicted_labels,labels)

        wandb.log({"loss": loss,"acc":acc})
        return loss

    def configure_optimizers(self):
        return torch.optim.Adam(self.parameters())

    def train_dataloader(self):
        return DataLoader(MovieLensTrainDataset(self.ratings, self.all_movieIds),
                          batch_size=512, num_workers=4)

# %% [markdown]
# We instantiate the NCF model using the class that we have defined above.

# %% [code]
num_users = ratings['userId'].max()+1
num_items = ratings['movieId'].max()+1

all_movieIds = ratings['movieId'].unique()

model = NCF(num_users, num_items, train_ratings, all_movieIds)

# %% [code]
wandb.watch(model)
early_stopping = EarlyStopping(
        monitor='loss',
        min_delta=0.00,
        patience=3,
        verbose=False,
        mode='min',
    )

trainer = pl.Trainer(max_epochs=100, gpus=1, reload_dataloaders_every_epoch=True,
                     progress_bar_refresh_rate=50, logger=False, checkpoint_callback=True,callbacks=[early_stopping])

trainer.fit(model)

# %% [markdown]
# ### Hit Ratio @ 10

# %% [code]
# User-item pairs for testing
test_user_item_set = set(zip(test_ratings['userId'], test_ratings['movieId']))

# Dict of all items that are interacted with by each user
user_interacted_items = ratings.groupby('userId')['movieId'].apply(list).to_dict()

hits = []
for (u,i) in tqdm(test_user_item_set):
    interacted_items = user_interacted_items[u]
    not_interacted_items = set(all_movieIds) - set(interacted_items)
    selected_not_interacted = list(np.random.choice(list(not_interacted_items), 99))
    test_items = selected_not_interacted + [i]

    predicted_labels = np.squeeze(model(torch.tensor([u]*100),
                                        torch.tensor(test_items)).detach().numpy())

    top10_items = [test_items[i] for i in np.argsort(predicted_labels)[::-1][0:10].tolist()]

    if i in top10_items:
        hits.append(1)
    else:
        hits.append(0)

print("The Hit Ratio @ 10 is {:.2f}".format(np.average(hits)))
wandb.log({"hit ratio": np.average(hits)})

error:
Traceback (most recent call last):
  File "main.py", line 359, in <module>
    trainer = pl.Trainer(max_epochs=100, gpus=1, reload_dataloaders_every_epoch=True,
  File "/home/max/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 499, in fit
    self.dispatch()
  File "/home/max/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 546, in dispatch
    self.accelerator.start_training(self)
  File "/home/max/.local/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 73, in start_training
    self.training_type_plugin.start_training(trainer)
  File "/home/max/.local/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 114, in start_training
    self._results = trainer.run_train()
  File "/home/max/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 637, in run_train
    self.train_loop.run_training_epoch()
  File "/home/max/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 492, in run_training_epoch
    batch_output = self.run_training_batch(batch, batch_idx, dataloader_idx)
  File "/home/max/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 654, in run_training_batch
    self.optimizer_step(optimizer, opt_idx, batch_idx, train_step_and_backward_closure)
  File "/home/max/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 425, in optimizer_step
    model_ref.optimizer_step(
  File "/home/max/.local/lib/python3.8/site-packages/pytorch_lightning/core/lightning.py", line 1390, in optimizer_step
    optimizer.step(closure=optimizer_closure)
  File "/home/max/.local/lib/python3.8/site-packages/pytorch_lightning/core/optimizer.py", line 214, in step
    self.__optimizer_step(*args, closure=closure, profiler_name=profiler_name, **kwargs)
  File "/home/max/.local/lib/python3.8/site-packages/pytorch_lightning/core/optimizer.py", line 134, in __optimizer_step
    trainer.accelerator.optimizer_step(optimizer, self._optimizer_idx, lambda_closure=closure, **kwargs)
  File "/home/max/.local/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 277, in optimizer_step
    self.run_optimizer_step(optimizer, opt_idx, lambda_closure, **kwargs)
  File "/home/max/.local/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 282, in run_optimizer_step
    self.training_type_plugin.optimizer_step(optimizer, lambda_closure=lambda_closure, **kwargs)
  File "/home/max/.local/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 163, in optimizer_step
    optimizer.step(closure=lambda_closure, **kwargs)
  File "/home/max/.local/lib/python3.8/site-packages/torch/optim/optimizer.py", line 89, in wrapper
    return func(*args, **kwargs)
  File "/home/max/.local/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/home/max/.local/lib/python3.8/site-packages/torch/optim/adam.py", line 66, in step
    loss = closure()
  File "/home/max/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 648, in train_step_and_backward_closure
    result = self.training_step_and_backward(
  File "/home/max/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 742, in training_step_and_backward
    result = self.training_step(split_batch, batch_idx, opt_idx, hiddens)
  File "/home/max/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 293, in training_step
    training_step_output = self.trainer.accelerator.training_step(args)
  File "/home/max/.local/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 156, in training_step
    return self.training_type_plugin.training_step(*args)
  File "/home/max/.local/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 125, in training_step
    return self.lightning_module.training_step(*args, **kwargs)
  File "main.py", line 318, in training_step
    print(type(labels))
  File "/home/max/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/max/.local/lib/python3.8/site-packages/torchmetrics/metric.py", line 152, in forward
    self.update(*args, **kwargs)
  File "/home/max/.local/lib/python3.8/site-packages/torchmetrics/metric.py", line 199, in wrapped_func
    return update(*args, **kwargs)
  File "/home/max/.local/lib/python3.8/site-packages/torchmetrics/classification/accuracy.py", line 142, in update
    self.correct += correct
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

Output:
I am explaining it here,
This command:
print(next(model.parameters()).device)

It will print the device on which your model's parameters are loaded.
To check if they are loaded on GPU or not, you can do this:
print(next(model.parameters()).is_cuda)

It will return a boolean value,
After seeing your code, and as you mentioned it was returning "CPU" when printed:
next(model.parameters()).device
Means that your model's parameter are loaded on CPU, but this line
trainer = pl.Trainer(max_epochs=100, gpus=1, reload_dataloaders_every_epoch=True,
                 progress_bar_refresh_rate=50, logger=False, checkpoint_callback=True,callbacks=[early_stopping])

Here gpus=1 means it will set the number of gpus to train,
since all your tensors are loaded on by default CPU, you were getting that error.
When you have set gpus=None it is no more using gpus to train.
To run on GPUs:
You have to move tensors from CPU to GPU,
For example:
ex_tensor=torch.zeros((7,7))
ex_tensor = ex_tensor.cuda()

And also your parameters of the model,
model = model.cuda()


Instruction: torchvision.transforms.RandomRotation fill argument not working

Input:
I am working on tensors and want to rotate them with torchvision.transforms.RandomRotation and use the fill option.
import torch
import torchvision

img1 = torch.rand((1, 16, 16))
img2 = torchvision.transforms.RandomRotation(45, fill=1)(img1)

However, I always get:

Argument fill/fillcolor is not supported for Tensor input. Fill value is zero

and it gets not filled with ones. I have the same problem with torchvision.transforms.RandomPerspective.
I am using Python 3.8 and PyTorch 1.7.1. I tried using fill=(1,), which seemed to be a workaround, but it did not worked for me. Do you know what could be wrong?

Output:
You're probably using torchvision v0.8.2 or older. This issue was fixed 5 months ago in the PR #2904. If you are not using v0.9.0 or newer, you won't be able to use fill in a Tensor input :(
So, the only solution is to upgrade your torchvision.


Instruction: Pytorch on GCP: Machine type is not available on this endpoint

Input:
I'm new to GCP so pardon for prehaps asking/missing something obvious here.
I'm trying to deploy and create a version resource on GCP with a custom pytorch model. Everything have been working fine until now until I try to create the a new version of the model. Then I keep getting:
INVALID_ARGUMENT: Machine type is not available on this endpoint.
I've tried switching between different types from their list here without luck. What am I missing?
Here's the script I run to deploy:
MODEL_NAME='test_iris'
MODEL_VERSION='v1'
RUNTIME_VERSION='2.4'
MODEL_CLASS='model.PyTorchIrisClassifier'
PYTORCH_PACKAGE='gs://${BUCKET_NAME}/packages/torch-1.8.1+cpu-cp37-cp37m-linux_x86_64.whl'

DIST_PACKAGE='gs://${BUCKET_NAME}/models/Test_model-0.1.tar.gz'
GCS_MODEL_DIR='models/'
REGION="europe-west1"

# Creating model on AI platform
gcloud alpha ai-platform models create ${MODEL_NAME}\
--region=europe-west1 --enable-logging \
--enable-console-logging

gcloud beta ai-platform versions create ${MODEL_VERSION} --model=${MODEL_NAME} \
    --origin=gs://${BUCKET_NAME}/${GCS_MODEL_DIR} \
    --python-version=3.7 \
    --machine-type=mls1-c4-m2\
    --runtime-version=${RUNTIME_VERSION} \
    --package-uris=${DIST_PACKAGE},${PYTORCH_PACKAGE} \
    --prediction-class=${MODEL_CLASS}

Thanks!

Output:
According to the documentation, you can only deploy a Custom prediction routine when using a legacy (MLS1) machine type for your model version. However, you can not use a regional endpoint with this type of machine, as stated here,

Regional endpoints only support Compute Engine (N1) machine types. You cannot use legacy (MLS1) machine types on regional endpoints.

As I can see, you have specified a regional endpoint with  the --region flag, which does not support the machine type you required for your use case. Thus, you need to change the model and its version to a global endpoint, so you won't face the error anymore.
In addition, when you specify a regional endpoint within gcloud create model --region, you need to specify the same region when creating the model's version. On the other hand, when creating a model in the global endpoint gcloud create model --regions, you can omit the region flag in the command gcloud ai-platform versions create. Note that the --regions command is used only for the global endpoint
Lastly, I must point out that, as per documentation, when selecting a region for the global endpoint, using the --regions flag when creating the model, your prediction nodes run in the specified region. Although, the AI Platform Prediction infrastructure managing your resources might not necessarily run in the same region.


Instruction: How to add extra dense layer on top of BertForSequenceClassification?

Input: I want to add an extra layer (and dropout) before the classification layer (I'm using PyTorch lightning) What is the best way to do it?

Output:
The class BertForSequenceClassification (that comes from the Huggingface Transformers when using PyTorch lightning) implements a fixed architecture. If you want to change it (e.g., by adding layers), you need to inherit your own module.
This is actually quite simple. You can copy the code of BertForSequenceClassification and modify the code between getting the pooled BERT output and getting the logits.
Note however that adding a hidden layer to a classifier does not make much difference when finetuning BERT. The capacity of the additional hidden layer is negligible compared to the entire stack of BERT layers. Even If you cannot finetune the entire model, fine-tuning just the last BERT layer is probably better than adding an extra layer to the classifier.


Instruction: Convert one-hot encoded dimension into the index of position of 1

Input:
I have a tensor of three dimensions [batch_size, sequence_length, number_of_tokens].
The last dimension is one-hot encoded. I want to receive a tensor of two dimensions, where sequence_length consists of the index position of '1' of the number_of_tokens dimension.
For example, to turn a tensor of shape (2, 3, 4):
[[[0, 1, 0, 0]
[1, 0, 0, 0]
[0, 0, 0, 1]]
[[1, 0, 0, 0]
[1, 0, 0, 0]
[0, 0, 1, 0]]]

into a tensor of shape (2, 3) where number_of_tokens dimension is converted into the 1's position:
[[1, 0, 3]
[0, 0, 2]]

I'm doing it to prepare the model result to compare to reference answer when computing loss, I hope it is correct way.

Output:
If your original tensor is as specified in your previous question, you can bypass the one-hot encoding and directly use the argmax:
t = torch.rand(2, 3, 4)
t = t.argmax(dim=2)


Instruction: How to change the value of a torch tensor with requires_grad=True so that the backpropogation can start again?

Input:
a=torch.tensor([1,2],dtype=dtype,requires_grad=True)
b=a[0]*a[1]
b.backward()
print(a.grad)

but when I use a+=1 and want to give a another value and do another round of backpropogation, it shows that a leaf Variable that requires grad is being used in an in-place operation.
However a=a+1 seems right. What's the difference between a=a+1 and a+=1?
(solved)
I find my doubt about torch actually lies in the following small example,
a=torch.tensor([1,2],dtype=dtype,requires_grad=True)
b=a[0]*a[1]
b.backward()
print(a.grad)

print(a.requires_grad)
c=2*a[0]*a[1]
c.backward()
print(a.grad)

When I do the first round of backpropogation, I get a.grad=[2,1].
Then I want to make a fresh start and calculate the differential of c with respect to a, however, the gradient seems to accumulate. How can I eliminate this effect?

Output:
The += operator is for in-place operations, while a = a+1 is not in-place (that is, a refers to a different tensor after this operation).
But in your example you don't seem to be using either of these, so it is hard to say what you want to achieve.


Instruction: Can we have inputs that is more than 1D in Pytorch (e.g word-embedding)

Input:
Say I have some text and I want to classify them into three groups food, sports, science. If I have a sentence I dont like to each mushrooms we can use wordembedding (say 100 dimensions) to create a 6x100 matrix for this particular sentense.
Ususally when training a neural-network our data is a 2D array with the dimensions n_obs x m_features
If I want to train a neural network on wordembedded sentences(i'm using Pytorch) then our input is 3D n_obs x (m_sentences x k_words)
e.g
#Say word-embedding is 3-dimensions
I = [1,2,3]
dont = [4,5,6]
eat = [7,8,9]
mushrooms = [10,11,12]

"I dont eat mushrooms" = [I,dont,eat,mushrooms] #First observation

Is the best way, when we have N>2 dimensions, to do some kind of pooling e.g mean, or can we use the actual 2D-features as input?

Output:
Technically the input will be 1D, but that doesn't matter.
The internal architecture of your neural network will take care of recognizing the different words. You could for example have a convolution with a stride equal to the embedding size.
You can flatten a 2D input to become 1D and it will work fine. This is the way you'd normally do it with word embeddings.
I = [1,2,3]
dont = [4,5,6]
eat = [7,8,9]
mushrooms = [10,11,12]

input = np.array([I,dont,eat,mushrooms]).flatten()

The inputs of a neural network have to always be of the same size, but as sentences are not, you will probably have to limit the the max length of the sentence to a set length of words and add paddings to the end of the shorter sentences:
I = [1,2,3]
Am = [4,5,6]
short = [7,8,9]
paddingword = [1,1,1]

input = np.array([I,Am,eat,short, paddingword]).flatten()

Also you might want to look at doc2vec from gensim, which is an easy way to make embeddings for texts, which are then easy to use for a text classification problem.


Instruction: Implementing Backprop for custom loss functions

Input:
I have a neural network Network that has a vector output. Instead of using a typical loss function, I would like to implement my own loss function that is a method in some class. This looks something like:
class whatever:
    def __init__(self, network, optimizer):
         self.network = network
         self.optimizer = optimizer

    def cost_function(relevant_data):
        ...implementation of cost function with respect to output of network and relevant_data...

    def train(self, epochs, other_params):
        ...part I'm having trouble with...

The main thing I'm concerned with is about taking gradients. Since I'm taking my own custom loss function, do I need to implement my own gradient with respect to the cost function?
Once I do the math, I realize that if the cost is J, then the gradient of J is a fairly simple function in terms of the gradient of the final layer of the Network. I.e, it looks something like: Equation link.
If I used some traditional loss function like CrossEntropy, my backprocess would look like:
objective = nn.CrossEntropyLoss()
for epochs:
    optimizer.zero_grad()
    output = Network(input)
    loss = objective(output, data)
    loss.backward()
    optimizer.step()

But how do we do this in my case? My guess is something like:
for epochs:
     optimizer.zero_grad()
     output = Network(input)
     loss = cost_function(output, data)
     #And here is where the problem comes in
     loss.backward()
     optimizer.step()

loss.backward() as I understand it, takes the gradients of the loss function with respect to the parameters. But can I still invoke it while using my own loss function (presumably the program doesn't know what the gradient equation is). Do I have to implement another method/subroutine to find the gradients as well?
Which brings me to my other question: if I do want to implement gradient calculation for my loss function, I also need the gradient of the neural network parameters. How do I obtain those? Is there a function for that?

Output:
As long as all your steps starting from the input till the loss function involve differentiable operations on PyTorch's tensors, you need not do anything extra. PyTorch builds a computational graph that keeps track of each operation, its inputs, and gradients. So, calling loss.backward() on your custom loss would still propagate gradients back correctly through the graph. A Gentle Introduction to torch.autograd from the PyTorch tutorials may be a useful reference.
After the backward pass, if you need to directly access the gradients for further processing, you can do so using the .grad attribute (so t.grad for tensor t in the graph).
Finally, if you have a specific use case for finding the gradient of an arbitrary differentiable function implemented using PyTorch's tensors with respect to one of its inputs (e.g. gradient of the loss with respect to a particular weight in the network), you could use torch.autograd.grad.


Instruction: PyTorch - Import dataset with images as labels

Input:
I have a dataset containing images as inputs and labels/targets as images as well. The structure in the folder is as follows:
> DATASET/
> ---TRAIN/
> ------image_xx.png
> ------label_xx.png
> ---TEST/
> ------image_xx.png
> ------label_xx.png

I've currently tried to use "ImageFolder" from torchvisions datasets to load the images as follows:
TRAIN_PATH = '/path/to/dataset/DATASET'
train_data = datasets.ImageFolder(root=TRAIN_PATH, transform=transforms.ToTensor())
train_loader = DataLoader(train_data, batch_size=16, shuffle=True)

However as shown below:
for img, label in train_loader:
    print(img.shape)
    print(label.shape)
    break

torch.Size([16, 3, 128, 128])
torch.Size([16])

The labels aren't images but rather indicies or something similar. Is there a convenient way of importing this dataset with the aforementioned structure?

Output:
The ImageFolder dataset is suitable when you have discrete, scalar classes for each image. It expects the directory structure to be such that each subdirectory contains a certain class.
For your case, you can simply define your own subclass of torch.nn.Dataset. This tutorial may be helpful.
A simple example (I have not tried running it to see if it works correctly):
import torch
import os
import cv2

class MyDataset(torch.utils.data.Dataset):

    def __init__(self, root_path, transform=None):
        self.data_paths = [f for f in sorted(os.listdir(root_path)) if f.startswith("image")]
        self.label_paths = [f for f in sorted(os.listdir(root_path)) if f.startswith("label")]
        self.transform = transform

    def __getitem__(self, idx):
        img = cv2.imread(self.data_paths[idx])
        label = cv2.imread(self.label_paths[idx])
        if self.transform:
            img = self.transform(img)
        return img, label

    def __len__(self):
        return len(self.data_paths)

TRAIN_PATH = '/path/to/dataset/DATASET/TRAIN/'
train_data = MyDataset(root_path=TRAIN_PATH, transform=transforms.ToTensor())
train_loader = torch.utils.data.DataLoader(train_data, batch_size=16, shuffle=True)


Instruction: difference between initializing bias by nn.init.constant and (bias=1) in pytorch

Input:
I'm making a code for AlexNet and i'm confused with how to initialize the weights
what is the difference between:
        for layer in self.cnnnet:
            if isinstance(layer, nn.Conv2d):
                 nn.init.constant_(layer.bias, 0)

and
nn.Linear(shape, bias=0)

Output:
The method nn.init.constant_ receives a parameter to initialize and a constant value to initialize it with. In your case, you use it to initialize the bias parameter of a convolution layer with the value 0.
The method nn.Linear the bias parameter is a boolean stating weather you want the layer to have a bias or not. By setting it to be 0, you're actually creating a linear layer with no bias at all.
A good practice is to start with PyTorch's default initialization techniques for each layer. This is done by just creating the layers, pytorch initializes them implicitly. In more advanced development stages you can also explicitly change it if necessary.
For more info see the official documentation of nn.Linear and nn.Conv2d.


Instruction: How to use torch.utils.tensorboard.SummaryWriter from the last interrupted events

Input:
As in this picture, if I want to add scalar from events.out.tfevents, but not create a new one.
How can I set the params int this code:
SummaryWriter(self, log_dir=None, comment='', purge_step=None, max_queue=10,
             flush_secs=120, filename_suffix='')

Output:
You should be able to run it the same way (e.g. log_dir has to be the same, tensorboard in your case).
You have to remember to use next global step when adding scalar though.
First run, assume it crashed at 9th step:
from torch.utils.tensorboard import SummaryWriter

writer = SummaryWriter("my_dir")
x = range(10)
for i in x:
    writer.add_scalar("y=x", i, i)
writer.close()

If you want to continue writing to this event file, you have to move last parameter global step by 10:
from torch.utils.tensorboard import SummaryWriter

writer = SummaryWriter("my_dir")
x = range(10)
for i in x:
    writer.add_scalar("y=x", i, i + 10) # start from step 10
writer.close()

Running first file, followed by the second one and opening tensorboard via tensorboard --logdir my_dir would give you:


Instruction: Tracing Tensor Sizes in TorchScript

Input:
I'm exporting a PyTorch model via TorchScript tracing, but I'm facing issues. Specifically, I have to perform some operations on tensor sizes, but the JIT compilers hardcodes the variable shapes as constants, braking compatibility with tensor of different sizes.
For example, create the class:
class Foo(nn.Module):
    """Toy class that plays with tensor shape to showcase tracing issue.

    It creates a new tensor with the same shape as the input one, except
    for the last dimension, which is doubled. This new tensor is filled
    based on the values of the input.
    """
    def __init__(self):
        nn.Module.__init__(self)

    def forward(self, x):
        new_shape = (x.shape[0], 2*x.shape[1])  # incriminated instruction
        x2 = torch.empty(size=new_shape)
        x2[:, ::2] = x
        x2[:, 1::2] = x + 1
        return x2

and run the test code:
x = torch.randn((3, 5))  # create example input

foo = Foo()
traced_foo = torch.jit.trace(foo, x)  # trace
print(traced_foo(x).shape)  # obviously this works
print(traced_foo(x[:, :4]).shape)  # but fails with a different shape!

I could solve the issue by scripting, but in this case I really need to use tracing. Moreover, I think that tracing should be able to handle tensor size manipulations correctly.

Output:
but in this case I really need to use tracing

You can freely mix torch.script and torch.jit wherever needed. For example one could do this:
import torch

class MySuperModel(torch.nn.Module):
    def __init__(self, *args, **kwargs):
        super().__init__()
        self.scripted = torch.jit.script(Foo(*args, **kwargs))
        self.traced = Bar(*args, **kwargs)

    def forward(self, data):
        return self.scripted(self.traced(data))

model = MySuperModel()
torch.jit.trace(model, (input1, input2))

You could also move part of the functionality dependent on shape to separate function and decorate it with @torch.jit.script:
@torch.jit.script
def _forward_impl(x):
    new_shape = (x.shape[0], 2*x.shape[1])  # incriminated instruction
    x2 = torch.empty(size=new_shape)
    x2[:, ::2] = x
    x2[:, 1::2] = x + 1
    return x2

class Foo(nn.Module):
    def forward(self, x):
        return _forward_impl(x)

There is no other way than script for that as it has to understand your code. With tracing it merely records operations you perform on the tensor and has no knowledge of control flow dependent on data (or shape of data).
Anyway, this should cover most of the cases and if it doesn't you should be more specific.


Instruction: Does dissecting a Pytorch model lower memory usage?

Input:
Suppose I have a Pytorch autoencoder model defined as:
class ae(torch.nn.Module):
   def __init__(self, z_dim, n_channel=3, size_=8):
       super(ae, self).__init__()
       self.encoder = Encoder()
       self.decoder = Decoder()

   def forward(self, x):
       z = self.encoder(x)
       x_reconstructed = self.decoder(z)
       return z, x_reconstructe

Now instead of defining an specific ae model and loading it, I can use the Encoder and Decoder code directly in my code. I know the number of total parameters wouldn't change but here's my question: since these two models are now separated, is it possible that the code can run on lower ram/gpu-memory? Does separating them means they do not need to be loaded into memory at once?
(Note that autoencoder is just an example, My question is really about any models that consists of several sub-modules).

Output:
is it possible that the code can run on lower ram/gpu-memory?

The way you created it right now no, it isn't. If you instantiate it and move to device, something along those lines:
encoder = ...
decoder = ...
autoencoder = ae(encoder, decoder).to("cuda")

It will take, in total, decoder + encoder GPU memory when moved to the device and will be loaded to memory at once.
But, instead, you could do this:
inputs = ...
inputs = inputs.to("cuda")

encoder = ...
encoder.to("cuda")

output = encoder(inputs)

encoder.to("cpu")  # Free GPU memory

decoder = ...
decoder.to("cuda")  # Uses less in total

result = decoder(output)

You could wrap this idea in model (or function), still one would have to wait for parts of the network to be copied to GPU and your performance will be inferior (but GPU memory will be smaller).
Depending on where you instantiate the models RAM memory footprint could also be lower (Python will automatically destroy object in function scope), let's look at this option (no need for casting to cpu as the object will be automatically garbage collected as mentioned above):
def encode(inputs):
    encoder = ...
    encoder.to("cuda")

    results = encoder(inputs)
    return results

def decode(inputs):
    decoder = ...
    decoder.to("cuda")
    return decoder(inputs)

outputs = encode(inputs)
result = decode(outputs)


Instruction: How to build CUDA custom C++ extension for PyTorch without CUDA?

Input:
I was tasked with creating a CI workflow for building a PyTorch CUDA extension for this application. Up until now, the application was deployed by creating the target AWS VM with a CUDA GPU, pushing all the sources there and running setup.py, but instead I want to do the build in our CI system and deploy pre-built binaries to the production environment.
When running setup.py in the CI system, I get the error "No CUDA GPUs are available" - which is true, there are no CUDA GPUs in the CI system. Is there a way to just build the CUDA extension without a CUDA GPU available?
This is the error message:
gcc -pthread -shared -B /usr/local/miniconda/envs/build/compiler_compat -L/usr/local/miniconda/envs/build/lib -Wl,-rpath=/usr/local/miniconda/envs/build/lib -Wl,--no-as-needed -Wl,--sysroot=/ /app/my-app/build/temp.linux-x86_64-3.6/my-extension/my-module.o -L/usr/local/miniconda/envs/build/lib/python3.6/site-packages/torch/lib -lc10 -ltorch -ltorch_cpu -ltorch_python -o build/lib.linux-x86_64-3.6/my-extension/my-module.cpython-36m-x86_64-linux-gnu.so
building 'my-extension.my-module._cuda_ext' extension
creating /app/my-app/build/temp.linux-x86_64-3.6/my-extension/src
Traceback (most recent call last):
  File "setup.py", line 128, in <module>
    'build_ext': BuildExtension
  File "/usr/local/miniconda/envs/build/lib/python3.6/site-packages/setuptools/__init__.py", line 153, in setup
    return distutils.core.setup(**attrs)
  File "/usr/local/miniconda/envs/build/lib/python3.6/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/local/miniconda/envs/build/lib/python3.6/distutils/dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "/usr/local/miniconda/envs/build/lib/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/usr/local/miniconda/envs/build/lib/python3.6/site-packages/setuptools/command/build_ext.py", line 79, in run
    _build_ext.run(self)
  File "/usr/local/miniconda/envs/build/lib/python3.6/distutils/command/build_ext.py", line 339, in run
    self.build_extensions()
  File "/usr/local/miniconda/envs/build/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 653, in build_extensions
    build_ext.build_extensions(self)
  File "/usr/local/miniconda/envs/build/lib/python3.6/distutils/command/build_ext.py", line 448, in build_extensions
    self._build_extensions_serial()
  File "/usr/local/miniconda/envs/build/lib/python3.6/distutils/command/build_ext.py", line 473, in _build_extensions_serial
    self.build_extension(ext)
  File "/usr/local/miniconda/envs/build/lib/python3.6/site-packages/setuptools/command/build_ext.py", line 196, in build_extension
    _build_ext.build_extension(self, ext)
  File "/usr/local/miniconda/envs/build/lib/python3.6/distutils/command/build_ext.py", line 533, in build_extension
    depends=ext.depends)
  File "/usr/local/miniconda/envs/build/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 468, in unix_wrap_ninja_compile
    cuda_post_cflags = unix_cuda_flags(cuda_post_cflags)
  File "/usr/local/miniconda/envs/build/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 377, in unix_cuda_flags
    cflags + _get_cuda_arch_flags(cflags) +
  File "/usr/local/miniconda/envs/build/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 1407, in _get_cuda_arch_flags
    capability = torch.cuda.get_device_capability()
  File "/usr/local/miniconda/envs/build/lib/python3.6/site-packages/torch/cuda/__init__.py", line 291, in get_device_capability
    prop = get_device_properties(device)
  File "/usr/local/miniconda/envs/build/lib/python3.6/site-packages/torch/cuda/__init__.py", line 296, in get_device_properties
    _lazy_init()  # will define _get_device_properties
  File "/usr/local/miniconda/envs/build/lib/python3.6/site-packages/torch/cuda/__init__.py", line 172, in _lazy_init
    torch._C._cuda_init()
RuntimeError: No CUDA GPUs are available

I'm not very familiar with CUDA and only half proficient in Python (I'm here as the "ops" part of "devops").

Output:
It is not a complete solution, as I lack details to completely figure out a solution. But it should help you or your teammates.

So first based on the source code, it is not required to reach torch._C._cuda_init() if you have CUDA arch flags set.
This means the pytorch is trying to figure out the CUDA arch because it is not specified by the user.
Here is a related thread. As you can see, setting the TORCH_CUDA_ARCH_LIST environment to something that fits your environment should work for you.


Instruction: How to use numpy dataset in Pytorch Lightning

Input:
I want to make a dataset using NumPy and then want to train and test a simple model like 'linear, or logistic`.
I am trying to learn Pytorch Lightning. I have found a tutorial that we can use the NumPy dataset and can use uniform distribution here. As a newcomer, I am not getting the full idea, how can I do that!
My code is given below
import numpy as np
import pytorch_lightning as pl
from torch.utils.data import random_split, DataLoader, TensorDataset
import torch
from torch.autograd import Variable
from torchvision import transforms

np.random.seed(42)

device = 'cuda' if torch.cuda.is_available() else 'cpu'

class DataModuleClass(pl.LightningDataModule):
    def __init__(self):
        super().__init__()
        self.constant = 2
        self.batch_size = 10
        self.transform = transforms.Compose([
            transforms.ToTensor()
        ])

    def prepare_data(self):
        a = np.random.uniform(0, 500, 500)
        b = np.random.normal(0, self.constant, len(x))

        c = a + b
        X = np.transpose(np.array([a, b]))

        idx = np.arange(500)
        np.random.shuffle(idx)

        # Uses foirst 400 random indices for training
        train_idx = idx[:400]
        # Uses the remaining indices for validation
        val_idx = idx[400:]

        # Generate train and validation dataset
        x_train, y_train = X[train_idx], y[train_idx]
        x_val, y_val = X[val_idx], y[val_idx]

        # Converting numpy array to Tensor
        self.x_train_tensor = torch.from_numpy(x_train).float().to(device)
        self.y_train_tensor = torch.from_numpy(y_train).float().to(device)

        self.x_val_tensor = torch.from_numpy(x_val).float().to(device)
        self.y_val_tensor = torch.from_numpy(y_val).float().to(device)

        training_dataset = TensorDataset(self.x_train_tensor, self.y_train_tensor)

        validation_dataset = TensorDataset(self.x_val_tensor, self.y_val_tensor)

        return training_dataset, validation_dataset

    def train_dataloader(self):
        training_dataloader = prepare_data() # Most probably this is wrong way!!!
        return DataLoader(self.training_dataloader)

    def val_dataloader(self):
        validation_dataloader = prepare_data() # Most probably this is wrong way!!!
        return DataLoader(self.validation_dataloader)

    # def test_dataloader(self):

obj = DataModuleClass()
print(obj.prepare_data())

This part is done based on the answer given [Here, I want to take a and b as features and c as label or target variable.]
Now, how, can I pass the dataset into the `training and validation method?

Output:
You can get the data from prepare_data() or setup() both using the following code.
def prepare_data(self):
    a = np.random.uniform(0, 500, 500)
    b = np.random.normal(0, self.constant, len(a))

    c = a + b
    X = np.transpose(np.array([a, b]))

    # Converting numpy array to Tensor
    self.x_train_tensor = torch.from_numpy(X).float().to(device)
    self.y_train_tensor = torch.from_numpy(c).float().to(device)

    training_dataset = TensorDataset(self.x_train_tensor, self.y_train_tensor)

    self.training_dataset = training_dataset

def setup(self):
    data = self.training_dataset
    self.train_data, self.val_data = random_split(data, [400, 100])

def train_dataloader(self):
    return DataLoader(self.train_data)

def val_dataloader(self):
    return DataLoader(self.val_data)

You can split the dataset using random_split().


Instruction: Calling pytorch neural network forward() gives error "mat1 and mat2 shapes cannot be multiplied"

Input:
I have following code which defines simple neural network:
class MlpNN(nn.Module):

    def __init__(self, in_dim=301, hidden_dims=[500,500,500,500,500,30]):

        super(MlpNN, self).__init__()

        self.net = nn.Sequential()
        self.net.add_module("lin_0", nn.Linear(in_dim, hidden_dims[0]))
        self.net.add_module("relu_0", nn.ReLU())

        layer_id = 1
        for hidden_dim in hidden_dims[1:]:
            self.net.add_module("lin_"+str(layer_id), nn.Linear(hidden_dim, hidden_dim))
            self.net.add_module("relu_"+str(layer_id), nn.ReLU())
            layer_id += 1

    def forward(self, x):
        return self.net(x)

The network formed is:
MlpNN(
  (net): Sequential(
    (lin_0): Linear(in_features=301, out_features=500, bias=True)
    (relu_0): ReLU()
    (lin_1): Linear(in_features=500, out_features=500, bias=True)
    (relu_1): ReLU()
    (lin_2): Linear(in_features=500, out_features=500, bias=True)
    (relu_2): ReLU()
    (lin_3): Linear(in_features=500, out_features=500, bias=True)
    (relu_3): ReLU()
    (lin_4): Linear(in_features=500, out_features=500, bias=True)
    (relu_4): ReLU()
    (lin_5): Linear(in_features=30, out_features=30, bias=True)
    (relu_5): ReLU()
  )
)

while doing forward(), it gives me following error:
mat1 and mat2 shapes cannot be multiplied (1x500 and 30x30)

Whats wrong I am doing here? I know it must be stupid basic given that I am quite new machine learning.

Output:
In your network, your lin_4 layer has 500 output features, while your lin_5 layer has 30 input features. This causes a mismatch in the shapes. To fix this, you should ensure that the output features of a layer and the input features of the layer after it are the same. In your code, you could do that as:
class MlpNN(nn.Module):

    def __init__(self, in_dim=301, hidden_dims=[500,500,500,500,500,30]):

        super(MlpNN, self).__init__()

        self.net = nn.Sequential()
        self.net.add_module("lin_0", nn.Linear(in_dim, hidden_dims[0]))
        self.net.add_module("relu_0", nn.ReLU())

        for layer_id in range(1, len(hidden_dims)):
            self.net.add_module("lin_"+str(layer_id), nn.Linear(hidden_dims[layer_id-1], hidden_dims[layer_id]))
            self.net.add_module("relu_"+str(layer_id), nn.ReLU())

    def forward(self, x):
        return self.net(x)


Instruction: How to get dataset from prepare_data() to setup() in PyTorch Lightning

Input:
I made my own dataset using NumPy in the prepare_data() methods using the DataModules method of PyTorch Lightning. Now, I want to pass the data into the setup() method to split into training and validation.
import numpy as np
import pytorch_lightning as pl
from torch.utils.data import random_split, DataLoader, TensorDataset
import torch
from torch.autograd import Variable
from torchvision import transforms

np.random.seed(42)

device = 'cuda' if torch.cuda.is_available() else 'cpu'

class DataModuleClass(pl.LightningDataModule):
    def __init__(self):
        super().__init__()
        self.constant = 2
        self.batch_size = 10

    def prepare_data(self):
        a = np.random.uniform(0, 500, 500)
        b = np.random.normal(0, self.constant, len(a))

        c = a + b
        X = np.transpose(np.array([a, b]))

        # Converting numpy array to Tensor
        self.x_train_tensor = torch.from_numpy(X).float().to(device)
        self.y_train_tensor = torch.from_numpy(c).float().to(device)

        training_dataset = TensorDataset(self.x_train_tensor, self.y_train_tensor)

        return training_dataset

    def setup(self):
        data = # What I have to write to get the data from prepare_data()
        self.train_data, self.val_data = random_split(data, [400, 100])

    def train_dataloader(self):
        training_dataloader = setup() # Need to get the training data
        return DataLoader(self.training_dataloader)

    def val_dataloader(self):
        validation_dataloader = prepare_data() # Need to get the validation data
        return DataLoader(self.validation_dataloader)

obj = DataModuleClass()
print(obj.prepare_data())

Output:
The same answer as your previous question...
def prepare_data(self):
    a = np.random.uniform(0, 500, 500)
    b = np.random.normal(0, self.constant, len(a))

    c = a + b
    X = np.transpose(np.array([a, b]))

    # Converting numpy array to Tensor
    self.x_train_tensor = torch.from_numpy(X).float().to(device)
    self.y_train_tensor = torch.from_numpy(c).float().to(device)

    training_dataset = TensorDataset(self.x_train_tensor, self.y_train_tensor)

    self.training_dataset = training_dataset

def setup(self):
    data = self.training_dataset
    self.train_data, self.val_data = random_split(data, [400, 100])

def train_dataloader(self):
    return DataLoader(self.train_data)

def val_dataloader(self):
    return DataLoader(self.val_data)


Instruction: Train a neural network when the training has only the derivative of output wrt all inputs

Input: There is a scalar function F with 1000 inputs. I want to train a model to predict F given the inputs. However, in the training dataset, we only know the derivative of F with respect to each input, not the value of F itself. How I can construct a neural network with this limitation in tensorflow or pytorch?

Output:
I think you can use torch.autograd to compute the gradients, and then use them for the loss. You need:
(a) A trainable nn.Module to represent the (unknown) function F:
class UnknownF(nn.Module):
  def __init__(self, ...):
    # whatever combinations of linear layers and activations and whatever...

  def forward(self, x):
    # x is 1000 dim vector
    y = self.layers(x)
    # y is a _scalar_ output
    return y

model = UnknownF(...)  # instansiate the model of the unknown function

(b) Training data:
x = torch.randn(n, 1000, requires_grad=True)  # n examples of 1000-dim vectors
dy = torch.randn(n, 1000)  # the corresponding n-dim gradients of the n inputs

(c) An optimizer:
opt = torch.optim.SGD(model.parameters(), lr=0.1)

(d) Put it together:
criterion = nn.MSELoss()

for e in range(num_epochs):
  for i in range(n):
    # batch size = 1, pick one example
    x_ = x[i, :]
    dy_ = dy[i, :]
    opt.zero_grad()
    # predict the unknown output
    y_ = model(x_)
    # compute the gradients of the model using autograd:
    pred_dy_ = autograd.grad(y_, x_, create_graph=True)[0]
    # compute the loss between the model's gradients and the GT ones:
    loss = criterion(pred_dy_, dy_)
    loss.backward()
    opt.step()  # update model's parameters accordingly.


Instruction: PyTorch: compare three tensors?

Input:
I have three boolean mask tensors that I want to create a boolean mask that if the value matches in three tensors then it is 1, else 0.
I tried torch.where(A == B == C, 1, 0), but it doesn't seem to support such.

Output:
The torch.eq operator only supports binary tensor comparisons, hence you need to perform two comparisons:
(A==B) & (B==C)


Instruction: Normalising images using mean and std of a dataset

Input:
I used the following snippet to compute the mean and std of the images in the cityscapes dataset to normalise them:
def compute_mean_std(dataloader):

    pop_mean = []

    pop_std = []

    for i, (img,mask, rgb_mask) in enumerate(dataloader):

        numpy_image = img.cpu().numpy()

        batch_mean = np.mean(numpy_image,axis=(0,2,3))
        pop_mean.append(batch_mean)

        #print(batch_mean.shape)

        batch_std = np.mean(numpy_image, axis=(0,2,3))
        pop_std.append(batch_std)

        #print(batch_std.shape)

    pop_mean = np.array(pop_mean).mean(axis=0)
    pop_std = np.array(pop_std).std(axis=0)

    print(pop_mean.shape)
    print(pop_std.shape)

    return(pop_mean, pop_std)

This code gave me the following mean and std:

MEAN = [0.28660315, 0.32426634, 0.28302112]
STD = [0.00310452, 0.00292714, 0.00296411]

but when I computed the mean and std of images after normalisation using these mean and std, they are not close to 0 and 1.
Is this approach correct to compute mean and std over the whole dataset and normalising images?

Output:
Your formulas are not correct. You can't take the mean of the values of a batch and then the standard deviation of these means and expect it to be the standard deviation over the entire dataset. Try something like:
total = 0.0
totalsq = 0.0
count = 0

for data, *_ in dataloader:
    count += np.prod(data.shape)
    total += data.sum()
    totalsq += (data**2).sum()

mean = total/count
var = (totalsq/count) - (mean**2)
std = torch.sqrt(var)


Instruction: Custom Dataset not accepting argument in PyTorch

Input:
I am trying to create a custom Dataset in PyTorch using this dataset. It is of the shape (X, 785), X being number of samples and each row containing the label at index 0 and 784 pixel values. This is my code :
from torch.utils.data import Dataset
def SignMNISTDataset(Dataset):

  def __init__(self, csv_file_path, mode='Train'):
    self.labels = []
    self.pixels = []
    self.mode = mode

    data = pd.read_csv(csv_file_path).values
    if self.mode == 'Train':
      self.labels = data[:,0].tolist()
      print("Training labels acquired")

    for idx in range(len(self.labels)):
      self.pixels.append(data[idx][1:].tolist())

  def __len__(self):
    return len(self.labels)

  def __getitem__(self, idx):
    pixels = self.pixels[idx]
    if self.mode == 'Train':
      labels = self.labels[idx]
      content = {"pixels":pixels, "label":labels}
    else:
      content = {"pixels":pixels}
    return content

training_data = SignMNISTDataset('sign_mnist_train/sign_mnist_train.csv', 'Train')

On running, I get the following error :
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-46-0173199f8794> in <module>()
     27     return content
     28
---> 29 training_data = SignMNISTDataset('sign_mnist_train/sign_mnist_train.csv', 'Train')
     30 from torch.utils.data import DataLoader
     31

TypeError: SignMNISTDataset() takes 1 positional argument but 2 were given

Where exactly is this coming from? Is the mode argument somehow not being read during the object creation?
My end goal is to create a neural network for classifying sign characters, following this tutorial.
I tried explicitly mentioning the keyword mode during the object creation. This is what I got -
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-48-fd796c48dc67> in <module>()
     27     return content
     28
---> 29 training_data = SignMNISTDataset('sign_mnist_train/sign_mnist_train.csv', mode='Train')

TypeError: SignMNISTDataset() got an unexpected keyword argument 'mode'

Output:
Please use
class SignMNISTDataset(Dataset):

Instead of
def SignMNISTDataset(Dataset):


Instruction: PyTorch backward() on a tensor element affected by nan in other elements

Input:
Consider the following two examples:
x = torch.tensor(1., requires_grad=True)
y = torch.tensor(0., requires_grad=True)
z = torch.full((2, ), float("nan"))
z0 = x * y / y
z1 = x + y
print(z0, z1) # tensor(nan, grad_fn=<DivBackward0>) tensor(1., grad_fn=<AddBackward0>)
z1.backward()
print(x.grad) # tensor(1.)

x = torch.tensor(1., requires_grad=True)
y = torch.tensor(0., requires_grad=True)
z = torch.full((2, ), float("nan"))
z[0] = x * y / y
z[1] = x + y
print(z) # tensor([nan, 1.], grad_fn=<CopySlices>)
z[1].backward()
print(x.grad) # tensor(nan)

In example 1, z0 does not affect z1, and the backward() of z1 executes as expected and x.grad is not nan. However, in example 2, the backward() of z[1] seems to be affected by z[0], and x.grad is nan.
How do I prevent this (example 1 is desired behaviour)? Specifically I need to retain the nan in z[0] so adding epsilon to division does not help.

Output:
When indexing the tensor in the assignment, PyTorch accesses all elements of the tensor (it uses binary multiplicative masking under the hood to maintain differentiability) and this is where it is picking up the nan of the other element (since 0*nan -> nan).
We can see this in the computational graph:

torchviz.make_dot(z1, params={'x':x,'y':y})
torchviz.make_dot(z[1], params={'x':x,'y':y})

If you wish to avoid this behaviour, either mask the nan's, or do as you have done in the first example - separate these into two different objects.


Instruction: Building CUDA extension fails even after issue in code (deprecated AT_CHECK) is fixed

Input:
I'm trying to install neural_renderer. Unfortunately, the original implementation only supports Python 2.7+ and PyTorch 0.4.0, so I'm using a fork that includes some fixes for compatibility with torch 1.7 (here). The main issue was using AT_CHECK(), which was not compatible with newer versions of PyTorch, and was replaced with TORCH_CHECK().
After running pip install neural_renderer_pytorch on the fixed version, using a virtual environment, I get the output (which I truncated to just the error):
 /tmp/pip-install-[somestring]/neural-renderer-pytorch_[somelongstring]/neural_renderer/cuda/load_textures_cuda.cpp:15:23: error: ‘AT_CHECK’ was not declared in this scope; did you mean ‘DCHECK’?
       15 | #define CHECK_CUDA(x) AT_CHECK(x.type().is_cuda(), #x " must be a CUDA tensor")
          |                       ^~~~~~~~

with [somestring] and [somelongstring] being some alphanumeric strings that changed with each compilation.
It looks like AT_CHECK is still being used somewhere in the code, but I don't know where. I know this error is exactly what the fork fixed, so I assume the cpp file is still cached somewhere from a previous compilation. But I can't figure out where exactly. I'm sure I'm on branch pytorch1.7 and running pip in the right repository; with torch==1.7.0 installed.
What I've tried so far, to no avail:

run pip cache purge before attempting to install
running pip with --no-cache-dir
deleting the virtualenv I'm using and making a new one
deleting the entire repository and making a new one

This issue on GitHub suggested just using PyTorch 1.4.0. This worked (i.e. I created a Python3.7 environment and ran conda install pytorch==1.4.0 torchvision==0.5.0 cudatoolkit=10.0 -c pytorch, then installed), but my goal is still to compile it for PyTorch 1.7.0 (and a newer version of Python).

Output:
If you want to install the fork, you cannot use pip install neural_renderer_pytorch. This command installs the original one.
To install the fork, you have to clone it to your local machine and install it:
git clone https://github.com/ZhengZerong/neural_renderer
cd neural_renderer
pip install .

You can do it in just one go as well:
pip install git+https://github.com/ZhengZerong/neural_renderer.git

Don't forget to uninstall the original version first, or just start a new venv.


Instruction: AttributeError: 'DataModuleClass' object has no attribute 'training_dataset'

Input:
I am trying to learn PyTorch Lightning by writing a very simple DataModuleClass. After prepare_data(), and setup() I am trying to check these functions are working or not. So, I am trying to get the training and validation dataset from setup(). But I am getting an error
AttributeError: 'DataModuleClass' object has no attribute 'training_dataset'

Code
def prepare_data(self):
    x = np.random.uniform(0, 10, 10)
    e = np.random.normal(0, self.sigma, len(x))

    # Making target or labels
    y = x + e

    # Marging x and e for 2 features
    X = np.transpose(np.array([x, e]))

    # Converting numpy array to Tensor
    self.x_train_tensor = torch.from_numpy(X).float().to(device)
    self.y_train_tensor = torch.from_numpy(y).float().to(device)

    training_dataset = TensorDataset(self.x_train_tensor, self.y_train_tensor)

    self.training_dataset = training_dataset

def setup(self):
    data = self.training_dataset
    self.train_data, self.val_data = random_split(data, [8, 2])
    return self.train_data, self.val_data

def train_dataloader(self):
    return DataLoader(self.train_data)

def val_dataloader(self):
    return DataLoader(self.val_data)

obj = DataModuleClass()
print(obj.setup())

Could you tell me why I am getting this error?

Output:
From the way the code looks to me.
The variable self.training_dataset of the DataModuleClass is initiated in prepare_data and setup need it in the first line.
But you called setup without calling training_dataset.
If prepare_data is expected to be called every time you create a DataModuleClass object then it best to put prepare_data in __init__. Like
def __init__(self, other_params):
    ..... all your code previously in __init__
    self.prepare_data()  # put this in the last line of this function

But if you don't need that then you need to call prepare_data before setup
obj = DataModuleClass()
obj.prepare_data()
print(obj.setup())

Or put prepare_data in setup itself.
def setup(self):
    self.prepare_data()
    data = self.training_dataset
    self.train_data, self.val_data = random_split(data, [8, 2])
    return self.train_data, self.val_data

Edit 1: See the actual value of self.train_data and self.val_data
The objects returned from setup are torch.utils.data.dataset.Subset.
There are basically 2 ways to get the tensors.
1. Treat them like lists
train_data, val_data = obj.setup()
print(train_data[0])

2. Use for loop
train_data, val_data = obj.setup()
for data in train_data:
    print(data)

Note
I'm not sure weather you'll get the tensors or TensorDataset. If it's the latter then use the same trick again, like
train_data, val_data = obj.setup()
train_tensor_data = train_data[0]
print(train_tensor_data[0])


Instruction: Why is accuracy droping in the last batch?

Input:
I am using PyTorch for a classification task. For some reason, the accuracy drops in the last iteration, I would like to know why? any answer is appreciated.
Hers is the code
class Classifier(nn.Module):
    def __init__(self):
        super(Classifier, self).__init__()
        self.layers = nn.Sequential(nn.Linear(89, 128),
                                    nn.ReLU(),
                                    nn.Linear(128, 64),
                                    nn.ReLU(),
                                    nn.Linear(64, 2))
    def forward(self, x):
        return self.layers(x)

def train(train_dl, model, epochs):
    loss_function = nn.CrossEntropyLoss()
    optimizer = optim.Adam(model.parameters(), lr=0.1)
    for epoch in range(epochs):
        for (features, target) in train_dl:
            optimizer.zero_grad()
            features, target = features.to(device), target.to(device)
            output = model(features.float())
            target = target.view(-1)
            loss = loss_function(output, target)
            loss.backward()
            optimizer.step()
            output = torch.argmax(output, dim=1)
            correct = (output == target).float().sum()
            accuracy = correct / 512
            print(accuracy, loss)
        break

model = Classifier().to(device)
train(train_dl, model, 10)

and the last part of the output
tensor(0.6465, device='cuda:0') tensor(0.6498, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.6348, device='cuda:0') tensor(0.6574, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.6582, device='cuda:0') tensor(0.6423, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.6660, device='cuda:0') tensor(0.6375, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.6719, device='cuda:0') tensor(0.6338, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.6426, device='cuda:0') tensor(0.6523, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.6699, device='cuda:0') tensor(0.6347, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.6582, device='cuda:0') tensor(0.6422, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.6543, device='cuda:0') tensor(0.6449, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.6465, device='cuda:0') tensor(0.6502, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.6992, device='cuda:0') tensor(0.6147, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.6777, device='cuda:0') tensor(0.6289, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.6836, device='cuda:0') tensor(0.6244, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.6738, device='cuda:0') tensor(0.6315, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.1387, device='cuda:0') tensor(0.5749, device='cuda:0', grad_fn=<NllLossBackward>)

Output:
Probably because your last batch size is less than 512. It would be better to change this line
accuracy = correct / 512

to:
accuracy = correct / features.shape[0]

Alternatively, if you don't want your last batch to have a different size, you can drop it when you create the DataLoader, by setting drop_last=True, something like this:
train_dl = DataLoader(..., drop_last=True)


Instruction: One liner tensor with conditions

Input:
I have two tensors:
a = torch.nn.Parameter(torch.rand(7, requires_grad=True))
b = torch.randint(0,60, (20,))

Is there a one liner (or a quick & short way) that can create a tensor (call it x) of size 20 (similar to "b") with conditions?
i.e.
[b<4 use a[0], 4 <=b<12 use a[1], 12<=b<22 use a[2], <28, <38, <50, >50] for every b

So if:
b = [12, 93, 54, 0...]

I want my new tensor "x" to be:
x = [a[2],a[6], a[6]...]

I'm going to use this "x" tensor to train and need the values the be backproped and learnable
i.e.
loss = torch.rand(20) * x

loss.backward() ...

So if one of the a's is not in x I want it to not change.

Output:
You can sum multiplicative masks of the conditions:
x = a[0]*(b<4) + a[1]*((4<=b)&(b<12)) + a[2]*((12<=b)&(b<22)) + a[3]*((22<=b)&(b<28)) + a[4]*((28<=b)&(b<30)) + a[5]*((30<=b)&(b<50)) + a[6]*(b>=50)