Instruction: Different Matrix multiplication behaviour between Keras and Pytorch

Input:
I was trying to understand how matrix multiplication works over 2 dimensions in DL frameworks and I stumbled upon an article here.
He used Keras to explain the same and it works for him.
But when I try to reproduce the same code in Pytorch, it fails with the error as in the output of the following code
Pytorch Code:
a = torch.ones((2,3,4))
b = torch.ones((7,4,5))
c = torch.matmul(a,b)
print(c.shape)

Output: RuntimeError: The size of tensor a (2) must match the size of tensor b (7) at non-singleton dimension 0

Keras Code:
a = K.ones((2,3,4))
b = K.ones((7,4,5))
c = K.dot(a,b)
print(c.shape)

Output:(2, 3, 7, 5)

Can somebody explain what is it that I'm doing wrong?

Output:
Matrix multiplication (aka matrix dot product) is a well defined algebraic operation taking two 2D matrices.
Deep-learning frameworks (e.g., tensorflow, keras, pytorch) are tuned to operate of batches of matrices, hence they usually implement batched matrix multiplication, that is, applying matrix dot product to a batch of 2D matrices.
The examples you linked to show how matmul processes a batch of matrices:
a = tf.ones((9, 8, 7, 4, 2))
b = tf.ones((9, 8, 7, 2, 5))
c = tf.matmul(a, b)

Note how all but last two dimensions are identical ((9, 8, 7)).
This is NOT the case in your example - the leading ("batch") dimensions are different, hence the error.
Using identical leading dimensions in pytorch:
a = torch.ones((2,3,4))
b = torch.ones((2,4,5))
c = torch.matmul(a,b)
print(c.shape)

results with

torch.Size([2, 3, 5])

If you insist on dot products with different batch dimensions, you will have to explicitly define how to multiply the two tensors. You can do that using the very flexible torch.einsum:
a = torch.ones((2,3,4))
b = torch.ones((7,4,5))
c = torch.einsum('ijk,lkm->ijlm', a, b)
print(c.shape)

Resulting with:

torch.Size([2, 3, 7, 5])


Instruction: Crossentropyloss Pytorch: Targetsize does not match Torchsize

Input:
I want to use the Crossentropyloss of pytorch but somehow my code only works with batchsize 2, so i am asuming there is something wrong with the shapes of target and output.
I get following error:
Value Error: Expected target size (50, 2), got torch.Size([50, 3])

My targetsize is (N=50,batchsize=3) and the output of my model is (N=50, batchsize=3, number of classes =2). Before the output layer my shape is (N=50,batchsize=3,dimensions=64).
How do i need to change the shapes so that the Crossentropyloss works?

Output:
Without further information about your model, here's what I would do. You have a many-to-many RNN which outputs (seq_len, batch_size, nb_classes) and the target is (seq_len, seq_len). The nn.CrossEntropyLoss module can take additional dimensions (batch_size, nb_classes, d1, d2, ..., dK) as an input.
You could make it work by permuting the axes, such that the outputted tensor is of shape (batch_size, nb_classes, seq_len). This should make it happen:
output = output.permute(0, 2, 1)

Additionally, your target will also have to change to be (batch_size, seq_len):
target = target.permute(1, 0)


Instruction: IndexError: index 2047 is out of bounds for axis 0 with size 1638

Input:
I want to train my dataset with training data and validation data.
Total data is 2048, train data is 1638, and validation data is 410 (20% of total).
Here are my codes

loading data (org: total training data)
 org_x = train_csv.drop(['id', 'digit', 'letter'], axis=1).values
 org_x = org_x.reshape(-1, 28, 28, 1)
 org_x = org_x/255
 org_x = np.array(org_x)
 org_x = org_x.reshape(-1, 1, 28, 28)
 org_x = torch.Tensor(org_x)

 x_test = test_csv.drop(['id','letter'], axis=1).values
 x_test = x_test.reshape(-1, 28, 28, 1)
 x_test = x_test/255
 x_test = np.array(x_test)
 x_test = x_test.reshape(-1, 1, 28, 28)
 x_test = torch.Tensor(x_test)

 y = train_csv['digit']
 y = list(y)
 print(len(y))
 org_y = np.zeros([len(y), 1])
 for i in range(len(y)):
   org_y[i] = y[i]
 org1 = np.array(org_y, dtype=object)

splitting data (org: total training data)
 from sklearn.model_selection import train_test_split

 x_train, x_valid, y_train, y_valid = train_test_split(
    org, org1, test_size=0.2, random_state=42)

transform
 transform = transforms.Compose([transforms.ToPILImage(),
                         transforms.ToTensor(),
                         transforms.Normalize((0.5, ), (0.5, )) ])

dataset
 class kmnistDataset(data.Dataset):
 def __init__(self, images, labels=None, transforms=None):
     self.x = images
     self.y = labels
     self.transforms = transforms

 def __len__(self):
     return (len(self.x))

 def __getitem__(self, idx):
     data = np.asarray(self.x[idx][0:]).astype(np.uint8)

     if self.transforms:
         data = self.transforms(data)

     if self.y is not None:
         return (data, self.y[i])
     else:
         return data

 train_data = kmnistDataset(x_train, y_train, transform)
 valid_data = kmnistDataset(x_valid, y_valid, transform)

 train_loader = DataLoader(train_data, batch_size=16, shuffle=True)
 valid_loader = DataLoader(valid_data, batch_size=16, shuffle = False)

I'll skip the model structure.

training(Here, I got the error message)
n_epochs = 30

valid_loss_min = np.Inf

for epoch in range(1, n_epochs+1):
    train_loss = 0
    valid_loss = 0

###################
# train the model #
###################
model.train()
for data in train_loader:
    inputs, labels = data[0], data[1]
    optimizer.zero_grad()
    output = model(inputs)
    loss = criterion(output, labels)
    loss.backward()
    optimizer.step()
    train_loss += loss.item()*data.size(0)

#####################
# validate the model#
#####################
model.eval()
for data in valid_loader:
    inputs, labels = data[0], data[1]
    output = model(inputs)
    loss = criterion(output, labels)
    valid_loss += loss.item()*data.size(0)

train_loss = train_loss/ len(train_loader.dataset)
valid_loss = valid_loss / len(valid_loader.dataset)

print('Epoch: {} \tTraining Loss: {:.6f} \tValidation Loss: {:.6f}'.format(
    epoch, train_loss, valid_loss))

Although I checked the data size, I got the error message below.

index 2047 is out of bounds for axis 0 with size 1638

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-42-b8783819421f> in <module>
     11     ###################
     12     model.train()
---> 13     for data in train_loader:
     14         inputs, labels = data[0], data[1]
     15         optimizer.zero_grad()

/opt/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py in __next__(self)
    433         if self._sampler_iter is None:
    434             self._reset()
--> 435         data = self._next_data()
    436         self._num_yielded += 1
    437         if self._dataset_kind == _DatasetKind.Iterable and \

/opt/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py in_next_data(self)
    473     def _next_data(self):
    474         index = self._next_index()  # may raise StopIteration
--> 475         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
    476         if self._pin_memory:
    477             data = _utils.pin_memory.pin_memory(data)

/opt/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py in fetch(self,
possibly_batched_index)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

/opt/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py in <listcomp(.0)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

<ipython-input-38-e5c87dd8a7ff> in __getitem__(self, idx)
     17
     18         if self.y is not None:
---> 19             return (data, self.y[i])
     20         else:
     21             return data

IndexError: index 2047 is out of bounds for axis 0 with size 1638

Can you explain why and how to solve it?

Output:
At first glance, you are using incorrect shapes: org_x = org_x.reshape(-1, 28, 28, 1). The channel axis you be the second one (unlike in TensorFlow), as (batch_size, channels, height, width):
org_x = org_x.reshape(-1, 1, 28, 28)

Same with x_test
x_test = x_test.reshape(-1, 1, 28, 28)

Also, you are accessing a list out of bound. You accessed self.y with i. Seems to me you should be returning (data, self.y[idx]) instead.


Instruction: pytorch - visualization of epoch versus error

Input:
I'm trying to learn pytorch, moving over from tensorflow.
Following a tutorial that used this code
import torch
import torch.nn as nn
import numpy as np
import matplotlib as plt
from sklearn import datasets
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split

# 0) Prepare data
bc = datasets.load_breast_cancer()
X, y = bc.data, bc.target

n_samples, n_features = X.shape

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1234)

# scale
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)

X_train = torch.from_numpy(X_train.astype(np.float32))
X_test = torch.from_numpy(X_test.astype(np.float32))
y_train = torch.from_numpy(y_train.astype(np.float32))
y_test = torch.from_numpy(y_test.astype(np.float32))

y_train = y_train.view(y_train.shape[0], 1)
y_test = y_test.view(y_test.shape[0], 1)

# 1) Model
# Linear model f = wx + b , sigmoid at the end
class Model(nn.Module):
    def __init__(self, n_input_features):
        super(Model, self).__init__()
        self.linear = nn.Linear(n_input_features, 1)

    def forward(self, x):
        y_pred = torch.sigmoid(self.linear(x))
        return y_pred

model = Model(n_features)

# 2) Loss and optimizer
num_epochs = 100
learning_rate = 0.01
criterion = nn.BCELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)

# 3) Training loop
for epoch in range(num_epochs):
    # Forward pass and loss
    y_pred = model(X_train)
    loss = criterion(y_pred, y_train)

    # Backward pass and update
    loss.backward()
    optimizer.step()

    # zero grad before new step
    optimizer.zero_grad()

    if (epoch+1) % 10 == 0:
        print(f'epoch: {epoch+1}, loss = {loss.item():.4f}')

with torch.no_grad():
    y_predicted = model(X_test)
    y_predicted_cls = y_predicted.round()
    acc = y_predicted_cls.eq(y_test).sum() / float(y_test.shape[0])
    print(f'accuracy: {acc.item():.4f}')

Trying to visualize the outputted model, and to visualize how similar the data input to output is, I'm having a bit of problems simply using matploblib,
predicted = model(X_test).detach().numpy()

plt.plot(X_test, y_train, 'ro')
plt.plot(X_test, y_predicted, 'b')
plt.show()

What is a good way to visualize this data? Thanks in advance,
using Ivan's answer - trying to add the train_acc to the function,
train_loss, valid_loss, valid_acc, train_acc = [], [], [],[]
for epoch in range(num_epochs):
    # Forward pass and loss
    y_pred = model(X_train)
    loss = criterion(y_pred, y_train)

    # Backward pass and update
    loss.backward()
    optimizer.step()

    # zero grad before new step
    optimizer.zero_grad()

    train_loss.append(loss.item())

#     acc_init = model(X_test).round().eq(y_train).sum() / float(y_train.shape[0])

    train_acc.append(acc_init)
    with torch.no_grad():
        y_pred = model(X_test)
        loss = criterion(y_pred, y_test)
        y_pred_cls = y_pred.round()
        acc = y_pred_cls.eq(y_test).sum() / float(y_test.shape[0])

        valid_loss.append(loss.item())
        valid_acc.append(acc.item())

    if (epoch+1) % 10 == 0:
        print(f'epoch: {epoch+1}, loss = {loss.item():.4f}')
        print(f'valid loss: {loss.item():.4f} valid acc: {acc.item():.4f}')

Output:
Simply collect the losses over epochs, then plot the values. You could compute the validation accuracy and its loss on every epoch, after you've updated the model. Something like:
train_loss, valid_loss, valid_acc = [], [], []
for epoch in range(num_epochs):
    # Forward pass and loss
    y_pred = model(X_train)
    loss = criterion(y_pred, y_train)

    # Backward pass and update
    loss.backward()
    optimizer.step()

    # zero grad before new step
    optimizer.zero_grad()

    train_loss.append(loss.item())

    with torch.no_grad():
        y_pred = model(X_test)
        loss = criterion(y_pred, y_test)
        y_pred_cls = y_pred.round()
        acc = y_pred_cls.eq(y_test).sum() / float(y_test.shape[0])

        valid_loss.append(loss.item())
        valid_acc.append(acc.item())

    if (epoch+1) % 10 == 0:
        print(f'epoch: {epoch+1}, loss = {loss.item():.4f}')
        print(f'valid loss: {loss.item():.4f} valid acc: {acc.item():.4f}')

Then plot with matplotlib or any other library:
plt.plot(train_loss, label='train_loss')
plt.plot(valid_loss, label='valid_loss')
plt.title('loss over epochs')
plt.legend()

Might as well compute the training accuracy!:
plt.plot(train_acc, label='train_acc')
plt.plot(valid_acc, label='valid_acc')
plt.title('accuracy over epochs')
plt.legend()


Instruction: How can I express this custom loss function in tensorflow?

Input:
I've got a loss function that fulfills my needs, but is only in PyTorch. I need to implement it into my TensorFlow code, but while most of it can trivially be "translated" I am stuck with a particular line:
y_hat[:, torch.arange(N), torch.arange(N)] = torch.finfo(y_hat.dtype).max  # to be "1" after sigmoid

You can see the whole code in following and it is indeed pretty straight forward except for that line:
def get_loss(y_hat, y):
 # No loss on diagonal
 B, N, _ = y_hat.shape
 y_hat[:, torch.arange(N), torch.arange(N)] = torch.finfo(y_hat.dtype).max  # to be "1" after sigmoid

 # calc loss
 loss = F.binary_cross_entropy_with_logits(y_hat, y)  # cross entropy

 y_hat = torch.sigmoid(y_hat)
 tp = (y_hat * y).sum(dim=(1, 2))
 fn = ((1. - y_hat) * y).sum(dim=(1, 2))
 fp = (y_hat * (1. - y)).sum(dim=(1, 2))
 loss = loss - ((2 * tp) / (2 * tp + fp + fn + 1e-10)).sum()  # fscore

return loss

So far I came up with following:
def get_loss(y_hat, y):
 loss = tf.keras.losses.BinaryCrossentropy()(y_hat,y)  # cross entropy (but no logits)

 y_hat = tf.math.sigmoid(y_hat)

 tp = tf.math.reduce_sum(tf.multiply(y_hat, y),[1,2])
 fn = tf.math.reduce_sum((y - tf.multiply(y_hat, y)),[1,2])
 fp = tf.math.reduce_sum((y_hat -tf.multiply(y_hat,y)),[1,2])
 loss = loss - ((2 * tp) / tf.math.reduce_sum((2 * tp + fp + fn + 1e-10)))  # fscore

return loss

so my questions boil down to:

What does torch.finfo() do and how to express it in TensorFlow?
Does y_hat.dtype just return the data type?

Output:
1. What does torch.finfo() do and how to express it in TensorFlow?
.finfo() provides a neat way to get machine limits for floating-point types. This function is available in Numpy, Torch as well as Tensorflow experimental.
.finfo().max returns the largest possible number representable as that dtype.
NOTE: There is also a .iinfo() for integer types.
Here are a few examples of finfo and iinfo in action.
print('FLOATS')
print('float16',torch.finfo(torch.float16).max)
print('float32',torch.finfo(torch.float32).max)
print('float64',torch.finfo(torch.float64).max)
print('')
print('INTEGERS')
print('int16',torch.iinfo(torch.int16).max)
print('int32',torch.iinfo(torch.int32).max)
print('int64',torch.iinfo(torch.int64).max)

FLOATS
float16 65504.0
float32 3.4028234663852886e+38
float64 1.7976931348623157e+308

INTEGERS
int16 32767
int32 2147483647
int64 9223372036854775807

If you want to implement this in tensorflow, you can use tf.experimental.numpy.finfo to solve this.
print(tf.experimental.numpy.finfo(tf.float32))
print('Max ->',tf.experimental.numpy.finfo(tf.float32).max)  #<---- THIS IS WHAT YOU WANT

Machine parameters for float32
---------------------------------------------------------------
precision =   6   resolution = 1.0000000e-06
machep =    -23   eps =        1.1920929e-07
negep =     -24   epsneg =     5.9604645e-08
minexp =   -126   tiny =       1.1754944e-38
maxexp =    128   max =        3.4028235e+38
nexp =        8   min =        -max
---------------------------------------------------------------

Max -> 3.4028235e+38

2. Does y_hat.dtype just return the data type?
YES.
In torch, it would return torch.float32 or something like that. In Tensorflow it would return tf.float32 or something like that.


Instruction: Pytorch CNN not learning

Input:
I am currently attempting to train a PyTorch CNN to classify demented and non-demented individuals based on MRI scans. However, during training, the loss of the model remains constant and the accuracy, while attempting to differentiate 3 classes, remains at .333. I have tried many of the suggestions offered by respondents to similar questions, yet none of them have worked for my specific task. These pieces of advice included changing the amount of convolutional units in the model, trying different loss functions, training the model on the raw dataset and then scaling up to a larger, augmented set of images, and altering parameters such as learning rate and batch size. I have attached my code and examples of input imagery below.
Image Examples
Healthy Brain
Mild Cognitive Impairment Brain
Alzheimer's Brain
Preprocessing Code
torch.cuda.set_device(0)
g = True
if g == True:
    for f in final_MRI_data:
        path = os.path.join(final_MRI_dir, f)
        matrix = nib.load(path)
        matrix.get_fdata()
        matrix = matrix.get_fdata()
        matrix.shape
        slice_ = matrix[90, :, :]
        img = Image.fromarray(slice_)
        img = img.crop((left, top, right, bottom))
        img = ImageOps.grayscale(img)
        data_matrices.append(img)

postda_data = []
for image in data_matrices:
    for i in range(30):
        transformed_img = transforms(image)
        transformed_img = np.asarray(transformed_img)
        postda_data.append(transformed_img)

final_MRI_labels = list(itertools.chain.from_iterable(itertools.repeat(x, 30) for x in
final_MRI_labels))

X = torch.Tensor(np.asarray([i for i in postda_data])).view(-1, 145, 200)
print(X.size())

y = torch.Tensor([i for i in final_MRI_labels]) #Target labels for cross entropy loss function

z = []
for val in final_MRI_labels:
    z.append(np.eye(3)[val])
z = torch.Tensor(np.asarray(z)) #Target one-hot encoded matrices for model testing function

Network Class
class Hl_Model(nn.Module):

    torch.cuda.set_device(0)

    def __init__(self):
        super().__init__()

        self.conv1 = nn.Conv2d(1, 32, 3, stride=2)
        self.conv2 = nn.Conv2d(32, 64, 3, stride=2)
        self.conv3 = nn.Conv2d(64, 128, 3, stride=2)
        self.conv4 = nn.Conv2d(128, 256, 3, stride=2)

        x = torch.randn(145,200).view(-1,1,145,200)
        self._to_linear = None
        self.convs(x)

        self.fc1 = nn.Linear(self._to_linear, 128, bias=True)
        self.fc2 = nn.Linear(128, 3)

    def convs(self, x):

        x = F.relu(self.conv1(x))
        x = F.relu(self.conv2(x))
        x = F.relu(self.conv3(x))
        x = F.max_pool2d(F.relu(self.conv4(x)), (2, 2), stride=2)

        if self._to_linear is None:
            self._to_linear = x[0].shape[0]*x[0].shape[1]*x[0].shape[2]
        return x

    def forward(self, x):
        x = self.convs(x)
        x = x.view(-1, self._to_linear)
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return F.softmax(x, dim=1)

Training Function
def train(net, train_fold_x, train_fold_y):

    optimizer = optim.Adam(net.parameters(), lr=0.05)
    BATCH_SIZE = 5
    EPOCHS = 50
    for epoch in range(EPOCHS):
        for i in tqdm(range(0, len(train_fold_x), BATCH_SIZE)):

            batch_x = train_fold_x[i:i+BATCH_SIZE].view(-1, 1, 145, 200)
            batch_y = train_fold_y[i:i+BATCH_SIZE]

            batch_x, batch_y = batch_x.to(device), batch_y.to(device)

            optimizer.zero_grad()
            outputs = net(batch_x)

            batch_y = batch_y.long()
            loss = loss_func(outputs, batch_y)

            loss.backward()
            optimizer.step()

        print(f"Epoch: {epoch} Loss: {loss}")

Testing Function
def test(net, test_fold_x, test_fold_y):

    test_fold_x.to(device)
    test_fold_y.to(device)

    correct = 0
    total = 0
    with torch.no_grad():
        for i in tqdm(range(len(test_fold_x))):
            real_class = torch.argmax(test_fold_y[i]).to(device)
            net_out = net(test_fold_x[i].view(-1, 1, 145, 200).to(device))
            pred_class = torch.argmax(net_out)

            if pred_class == real_class:
                correct += 1
            total +=1

Cross-Validation Loop
for i in range(6):
    result = next(skf.split(X, y))
    X_train = X[result[0]]
    X_test = X[result[1]]
    y_train = y[result[0]]
    y_test = z[result[1]]
    train(hl_model, X_train, y_train)
    test(hl_model, X_test, y_test)

Output During Training:
  0%|          | 0/188 [00:00<?, ?it/s]
  1%|          | 1/188 [00:01<05:35,  1.79s/it]
  5%|4         | 9/188 [00:01<03:45,  1.26s/it]
  9%|8         | 16/188 [00:02<02:32,  1.13it/s]
 12%|#2        | 23/188 [00:02<01:43,  1.60it/s]
 16%|#6        | 31/188 [00:02<01:09,  2.27it/s]
 21%|##        | 39/188 [00:02<00:46,  3.20it/s]
 25%|##5       | 47/188 [00:02<00:31,  4.49it/s]
 30%|##9       | 56/188 [00:02<00:21,  6.26it/s]
 35%|###4      | 65/188 [00:02<00:14,  8.67it/s]
 39%|###9      | 74/188 [00:02<00:09, 11.85it/s]
 44%|####4     | 83/188 [00:02<00:06, 15.92it/s]
 49%|####8     | 92/188 [00:02<00:04, 21.01it/s]
 54%|#####3    | 101/188 [00:03<00:03, 27.03it/s]
 59%|#####8    | 110/188 [00:03<00:02, 33.91it/s]
 63%|######3   | 119/188 [00:03<00:01, 41.21it/s]
 68%|######8   | 128/188 [00:03<00:01, 48.36it/s]
 73%|#######2  | 137/188 [00:03<00:00, 55.36it/s]
 78%|#######7  | 146/188 [00:03<00:00, 61.09it/s]
 82%|########2 | 155/188 [00:03<00:00, 65.87it/s]
 87%|########7 | 164/188 [00:03<00:00, 69.85it/s]
 92%|#########2| 173/188 [00:03<00:00, 72.93it/s]
 97%|#########6| 182/188 [00:04<00:00, 74.88it/s]
100%|##########| 188/188 [00:04<00:00, 45.32it/s]
Epoch: 0 Loss: 1.5514447689056396

  0%|          | 0/188 [00:00<?, ?it/s]
  5%|4         | 9/188 [00:00<00:02, 85.13it/s]
 10%|9         | 18/188 [00:00<00:02, 84.42it/s]
 14%|#4        | 27/188 [00:00<00:01, 83.22it/s]
 19%|#9        | 36/188 [00:00<00:01, 82.64it/s]
 24%|##3       | 45/188 [00:00<00:01, 82.23it/s]
 29%|##8       | 54/188 [00:00<00:01, 82.17it/s]
 34%|###3      | 63/188 [00:00<00:01, 82.13it/s]
 38%|###8      | 72/188 [00:00<00:01, 81.66it/s]
 43%|####2     | 80/188 [00:00<00:01, 79.76it/s]
 47%|####6     | 88/188 [00:01<00:01, 79.66it/s]
 52%|#####1    | 97/188 [00:01<00:01, 80.58it/s]
 56%|#####6    | 106/188 [00:01<00:01, 80.36it/s]
 61%|######1   | 115/188 [00:01<00:00, 80.64it/s]
 66%|######5   | 124/188 [00:01<00:00, 80.84it/s]
 71%|#######   | 133/188 [00:01<00:00, 80.54it/s]
 76%|#######5  | 142/188 [00:01<00:00, 80.98it/s]
 80%|########  | 151/188 [00:01<00:00, 80.86it/s]
 85%|########5 | 160/188 [00:01<00:00, 80.77it/s]
 90%|########9 | 169/188 [00:02<00:00, 78.81it/s]
 94%|#########4| 177/188 [00:02<00:00, 78.53it/s]
 98%|#########8| 185/188 [00:02<00:00, 77.88it/s]
100%|##########| 188/188 [00:02<00:00, 80.35it/s]
Epoch: 1 Loss: 1.5514447689056396

  0%|          | 0/188 [00:00<?, ?it/s]
  5%|4         | 9/188 [00:00<00:02, 83.56it/s]
 10%|9         | 18/188 [00:00<00:02, 82.41it/s]
 14%|#3        | 26/188 [00:00<00:01, 81.49it/s]
 19%|#8        | 35/188 [00:00<00:01, 81.65it/s]
 23%|##3       | 44/188 [00:00<00:01, 81.55it/s]
 28%|##7       | 52/188 [00:00<00:01, 80.41it/s]
 32%|###1      | 60/188 [00:00<00:01, 79.40it/s]
 37%|###6      | 69/188 [00:00<00:01, 80.17it/s]
 41%|####1     | 78/188 [00:00<00:01, 80.29it/s]
 46%|####6     | 87/188 [00:01<00:01, 80.81it/s]
 51%|#####1    | 96/188 [00:01<00:01, 80.95it/s]
 55%|#####5    | 104/188 [00:01<00:01, 80.24it/s]
 60%|######    | 113/188 [00:01<00:00, 80.56it/s]
 65%|######4   | 122/188 [00:01<00:00, 80.56it/s]
 70%|######9   | 131/188 [00:01<00:00, 80.78it/s]
 74%|#######4  | 140/188 [00:01<00:00, 79.65it/s]
 79%|#######9  | 149/188 [00:01<00:00, 80.14it/s]
 84%|########4 | 158/188 [00:01<00:00, 80.70it/s]
 89%|########8 | 167/188 [00:02<00:00, 80.88it/s]
 94%|#########3| 176/188 [00:02<00:00, 81.22it/s]
 98%|#########8| 185/188 [00:02<00:00, 81.03it/s]
100%|##########| 188/188 [00:02<00:00, 80.66it/s]
Epoch: 2 Loss: 1.5514447689056396

This output repeats all the way to "Epoch: 49 Loss: 1.5514447689056396"
Thanks in advance for any advice.

Output:
It seems that the problem is due to the softmax activation at the last step of model forward and your loss function, loss_func = nn.CrossEntropyLoss() which actually takes raw logits instead. Please check the official documentation:

class torch.nn.CrossEntropyLoss(weight: Optional[torch.Tensor] = None, size_average=None, ignore_index: int = -100, reduce=None, reduction: str = 'mean')

and

This criterion combines nn.LogSoftmax() and nn.NLLLoss() in one single class.
The input is expected to contain raw, unnormalized scores for each class.


Instruction: how many epoch for training 1k images

Input:
am doing training for detecting the objects using yolov3 but i face some problem when i set batch_size > 1 it causes me cuda out of memory so i searched in google to see another solution found it depends on my GPU (GTX 1070 8G) .
may be the number of epoch is  high and it require to be optimized .
maybe the epoch number should be decreased? and for training 1k images and 200 pic for validations .
what is best epoch should i set to avoid overfitting?

Output:
your model's overfitting wont depend on the no. of epochs you set.....
since you hav made a val split in your data, make sure that your train loss - val loss OR train acc - val acc is nearly the same.This will assure that your model is not overfitting


Instruction: "for tokens_tensor, segments_tensors, att_mask, pos_id, trg in data_loader: NameError: name 'data_loader' is not defined"

Input:
I am trying to implement question answering model with a BERT transformer implemented by jugapuff.
Link to the code: https://github.com/jugapuff/BERT-for-bAbi-task
After executing the main.py file which is written below as well, I m getting this error: "for tokens_tensor, segments_tensors, att_mask, pos_id, trg in data_loader: NameError: name 'data_loader' is not defined"
from dataloader import bAbi_Dataset
import torch
import torch.nn as nn
from model import model
from pytorch_transformers import AdamW

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

if torch.cuda.is_available():
    print("GPU:" + str(torch.cuda.get_device_name(0)))

my_model = model()
my_model.to(device)

optimizer = AdamW(my_model.parameters())
criterion = nn.NLLLoss()

EPOCHS = 10
for epoch in range(1, EPOCHS+1):

    my_model.train()

    train_loss = 0
    length = 0
    for tokens_tensor, segments_tensors, att_mask, pos_id, trg in data_loader:
        output = my_model(tokens_tensor.to(device), segments_tensors.to(device), att_mask.to(device), pos_id.to(device))
        loss = criterion(output, trg.to(device))
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        length+=1
        train_loss += loss.item()
        if length % 10 == 0:
          print("\t\t{:3}/25000 : {}".format(length, train_loss / length))

    epoch_loss = train_loss / length
    print("##################")
    print("{} epoch Loss : {:.4f}".format(epoch, epoch_loss))

and data_loader.py is as
import os
import torch
import torch.utils.data as data
from pytorch_transformers import BertTokenizer

def _parse( file, only_supporting=False):
        data, story = [], []
        for line in file:
            tid, text = line.rstrip('\n').split(' ', 1)
            if tid == '1':
                story = []
            if text.endswith('.'):
                story.append(text[:])
            else:
                query, answer, supporting = (x.strip() for x in text.split('\t'))
                if only_supporting:
                    substory = [story[int(i) - 1] for i in supporting.split()]
                else:
                    substory = [x for x in story if x]
                data.append((substory, query[:-1], answer))
                story.append("")
        return data

def build_trg_dics(tenK=True, path="tasks_1-20_v1-2", train=True):

    if tenK:
        dirname = os.path.join(path, 'en-10k')
    else:
        dirname = os.path.join(path, 'en')

    for (dirpath, dirnames, filenames) in os.walk(dirname):
        filenames = filenames

    if train:
        filenames = [filename for filename in filenames if  "train.txt" in filename]
    else:
        filenames = [filename for filename in filenames if  "test.txt" in filename]

    temp = []
    for filename in filenames:
        f = open(os.path.join(dirname, filename), 'r')
        parsed =_parse(f)
        temp.extend([d[2] for d in parsed])
    temp = set(temp)

    trg_word2id = {word:i for i, word in enumerate(temp)}
    trg_id2word = {i:word for i, word in enumerate(temp)}
    return trg_word2id, trg_id2word

class bAbi_Dataset(data.Dataset):

    def __init__(self, trg_word2id, tenK=True, path = "tasks_1-20_v1-2", train=True):
        # joint is Default

        self.tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

        if tenK:
            dirname = os.path.join(path, 'en-10k')
        else:
            dirname = os.path.join(path, 'en')

        for (dirpath, dirnames, filenames) in os.walk(dirname):
            filenames = filenames

        if train:
            filenames = [filename for filename in filenames if  "train.txt" in filename]
        else:
            filenames = [filename for filename in filenames if  "test.txt" in filename]

        self.src = []
        self.trg = []

        for filename in filenames:
            f = open(os.path.join(dirname, filename), 'r')
            parsed = _parse(f)
            self.src.extend([d[:2] for d in parsed])
            self.trg.extend([trg_word2id[d[2]] for d in parsed])
        self.trg = torch.tensor(self.trg)

    def __getitem__(self, index):
        src_seq = self.src[index]
        trg = self.trg[index]
        src_seq, seg_seq, att_mask, pos_id = self.preprocess_sequence(src_seq)

        return src_seq, seg_seq, att_mask, pos_id, trg

    def __len__(self):
        return len(self.trg)

    def preprocess_sequence(self, seq):

        text =  ["[CLS]"] + list(seq[0]) + ["[SEP]"] + [seq[1]] + ["[SEP]"]

        tokenized_text = self.tokenizer.tokenize(" ".join(text))
        indexed_text = self.tokenizer.convert_tokens_to_ids(tokenized_text)
        where_is_sep = indexed_text.index(102) + 1
        segment_ids = [0 ]* (where_is_sep) + [1] * (len(indexed_text)- where_is_sep)
        attention_mask = [1] *len(indexed_text)
        pos_id = [i for i in range(len(indexed_text))]

        return torch.tensor(indexed_text), torch.tensor(segment_ids), torch.tensor(attention_mask), torch.tensor(pos_id)

def collate_fn(data):
    def merge(sequences):
        lengths = [len(seq) for seq in sequences]
        padded_seqs = torch.zeros(len(sequences), 512).long()
        for i, seq in enumerate(sequences):

            end = lengths[i]
            if end <= 512:
                padded_seqs[i, :end] = seq[:end]
            else:
                padded_seqs[i] = seq[-512:]

        return padded_seqs

    def pos_merge(sequences):

        lengths = [len(seq) for seq in sequences]
        padded_seqs = torch.zeros(len(sequences), 512).long()
        for i, seq in enumerate(sequences):

            padded_seqs[i] = torch.tensor([i for i in range(512)])

        return padded_seqs

    src_seqs, seg_seqs, att_mask, pos_id, trgs = zip(*data)
    src_seqs = merge(src_seqs)
    seg_seqs = merge(seg_seqs)
    att_mask = merge(att_mask)
    pos_id = pos_merge(pos_id)
    trgs = torch.tensor(trgs)
    return src_seqs, seg_seqs, att_mask, pos_id, trgs

data_loader variable declaration in main.py is missing. So I tried to load data_loader as
for tokens_tensor, segments_tensors, att_mask, pos_id, trg in dataloader.collate_fn(bAbi_Dataset):
use collate_fn() function in data_loader.py, but it did not work. When I change it as above, it gives the following error:
Traceback (most recent call last):
  File "main.py", line 27, in <module>
  File "/content/BERT-for-bAbi-task/dataloader.py", line 133, in collate_fn
    src_seqs, seg_seqs, att_mask, pos_id, trgs = zip(*data)
  File "/usr/lib/python3.6/typing.py", line 682, in inner
    return func(*args, **kwds)
  File "/usr/lib/python3.6/typing.py", line 1107, in __getitem__
    params = tuple(_type_check(p, msg) for p in params)
  File "/usr/lib/python3.6/typing.py", line 1107, in <genexpr>
    params = tuple(_type_check(p, msg) for p in params)
  File "/usr/lib/python3.6/typing.py", line 374, in _type_check
    raise TypeError(msg + " Got %.100r." % (arg,))
TypeError: Parameters to generic types must be types. Got 0.

Could anyone please help me how to correct the error?

Output:
I will just give you some pointers:

collate_fn is not meant to be called with a dataset as argument. It is a special callback function given to a dataloader and used to collate batch elements into a batch.

Since bAbi_Dataset in /dataloader.py is defined as a torch.utils.data.Dataset I would guess you are meant to initialize it instead. It is defined here as:
def __init__(self, trg_word2id, tenK=True, path = "tasks_1-20_v1-2", train=True)

There is another function build_trg_dics in /dataloader.py which is used to create the parse the content from files. You should take a look at them before setting the right arguments for bAbi_Dataset.

Lastly, when you have your dataset initialized, you can attach a dataloader on it using torch.utils.data.DataLoader. This would look like:
data_loader = DataLoader(dataset, batch_size=16)

At this point, you might even need to plug in the collate function provided in /dataloader.py.

If you don't really know what you are doing, I would suggest you start with a working repository and work your way from there. Good luck!


Instruction: Torch, how to use Multiple GPU for different dataset

Input:
Assume that I have 4 different datasets and 4 GPU like below
4 dataset
dat0 = [np.array(...)], dat1 = [np.array(...)] , dat2 = [np.array(...)] , dat3 = [np.array(...)]

4 GPU
device = [torch.device(f'cuda:{i}') for i in range(torch.cuda.device_count())]

assume all the four data set have already converted into tensor and transfer to 4 different GPU.
Now, I have a function f from other module which can be used on GPU
How can I do the following at the same time,
compute 4 resulf of this
 ans0 = f(dat0) on device[0], ans1 = f(dat1) on device[1], ans2 = f(dat2) on device[2], ans3 = f(dat3) on device[3]

then move all the 4 ans back to cpu then calculate the sum
 ans = ans0 + ans1 + ans2 + ans3

Output:
Assuming you only need ans for inference. You can easily perform those operations but you will certainly need function f to be on all four GPUs at the same time.
Here is what I would try: duplicate f four times and send to each GPU. Then compute the intermediate result, sending back each result to the CPU for the final operation:
fns = [f.clone().to(device) for device in devices]

results = []
for fn, data in zip(fns, datasets):
    result = fn(data).detach().cpu()
    results.append(result)

ans = torch.stack(results).sum(dim=0)


Instruction: How does torchvision.transforms.Normalize operate?

Input:
I don't understand how the normalization in Pytorch works.
I want to set the mean to 0 and the standard deviation to 1 across all columns in a tensor x of shape (2, 2, 3).
A simple example:
>>> x = torch.tensor([[[ 1.,  2.,  3.],
                       [ 4.,  5.,  6.]],

                       [[ 7.,  8.,  9.],
                        [10., 11., 12.]]])

>>> norm = transforms.Normalize((0, 0), (1, 1))
>>> norm(x)
tensor([[[ 1.,  2.,  3.],
         [ 4.,  5.,  6.]],

        [[ 7.,  8.,  9.],
         [10., 11., 12.]]])

So nothing has changed when applying the normalization transform. Why is that?

Output:
To give an answer to your question, you've now realized that torchvision.transforms.Normalize doesn't work as you had anticipated. That's because it's not meant to:

normalize: (making your data range in [0, 1]) nor

standardize: making your data's mean=0 and std=1 (which is what you're looking for.

The operation performed by T.Normalize is merely a shift-scale transform:

output[channel] = (input[channel] - mean[channel]) / std[channel]

The parameters names mean and std which seems rather misleading knowing that it is not meant to refer to the desired output statistics but instead any arbitrary values. That's right, if you input mean=0 and std=1, it will give you output = (input - 0) / 1 = input. Hence the result you received where function norm had no effect on your tensor values when you were expecting to get a tensor of mean and variance 0 and 1, respectively.
However, providing the correct mean and std parameters, i.e. when mean=mean(data) and std=std(data), then you end up calculating the z-score of your data channel by channel, which is what is usually called 'standardization'. So in order to actually get mean=0 and std=1, you first need to compute the mean and standard deviation of your data.
If you do:
>>> mean, std = x.mean(), x.std()
(tensor(6.5000), tensor(3.6056))

It will give you the global average, and global standard deviation respectively.
Instead, what you want is to measure the 1st and 2nd order statistics per-channel. Therefore, we need to apply torch.mean and torch.std on all dimensions expect dim=1. Both of those functions can receive a tuple of dimensions:
>>> mean, std = x.mean((0,2)), x.std((0,2))
(tensor([5., 8.]), tensor([3.4059, 3.4059]))

The above is the correct mean and standard deviation of x measured along each channel. From there you can go ahead and use T.Normalize(mean, std) to correctly transform your data x with the correct shift-scale parameters.
>>> norm(x)
tensor([[[-1.5254, -1.2481, -0.9707],
         [-0.6934, -0.4160, -0.1387]],

        [[ 0.1387,  0.4160,  0.6934],
         [ 0.9707,  1.2481,  1.5254]]])


Instruction: Chose rows of 3d Tensor based on some repeated indices. Tricky Slicing

Input:
So this is a tricky bit of Tensor slicing I'm trying.
I have a tensor A which is 3d
>> A.shape
torch.Size([60, 10, 16])

So this tensor is composed of 5 different data samples, where in dim=0 we have cuts at split_ids = [10, 14, 10, 12, 14] i.e first 10 elements belong to sample1; next 14 belong to sample2 & so on.. I can split the tensor in such a way:
>> torch.split(A, split_ids, dim=0)
(tensor([[[-0.3888, -...Backward>), tensor([[[ 2.6473e-0...Backward>), tensor([[[ 1.1621,  ...Backward>), tensor([[[ 0.1953, -...Backward>), tensor([[[ 8.1993e-0...Backward>))

This comprises of tuple of 5 elements (or 5 tensors), of shapes Size(10,10,16);Size(14,10,16); and so on for the splits we had.
Now, comes the tricky part - I have another indices mapping that I have derived some previous processing for each of these individual splits. Its a list of 1d tensors like this:
>> reverse_map
[tensor([1, 2, 2, 1, ...='cuda:0'), tensor([ 7,  7,  9, ...='cuda:0'), tensor([7, 7, 4, 3, ...='cuda:0'), tensor([ 9,  4,  9, ...='cuda:0'), tensor([ 0,  0,  0, ...='cuda:0')]
>> reverse_map[0]
tensor([1, 2, 2, 1, 1, 2, 0, 1, 2, 0, 0, 1, 6, 1, 7, 1, 2, 7, 4, 5, 3, 4, 9, 7,
        7, 3, 8, 7, 7, 7], device='cuda:0')

So I need to basically use these indices and pull the count of these indices from the above split tensors i.e For tensor.Size(10,10,16) I need to pull [0, 0:3, :] which is for index 0 in dim=0, I need to pull 0:3 in dim=1 because there are 3 0's in the indexing. Then at index 1 I need to pull first 4 vectors since there are 4 1's.. and so on.
Whats the best way to do this ? Does scatter_() help here ?

Output:
I think a combination of torch.bincount and this answer can give you what you want.
For simplicity, let's focus on the first tensor split from A and the first reverse_map. You can then apply this code to the other splits and reverse_maps. Let source be the first split of shape (10, 10, 16).
Here's how it goes:
# inputs
source = torch.arange(10*10*16).view(10, 10, 16)
reverse_map = torch.tensor(tensor([1, 2, 2, 1, 1, 2, 0, 1, 2, 0, 0, 1, 6, 1, 7, 1, 2, 7, 4, 5, 3, 4, 9, 7, 7, 3, 8, 7, 7, 7])

# how many columns to pull for each row - use bincount to find out!
lengths = torch.bincount(reverse_map, minlength=source.shape[0])

# use a mask to pull the elements
mask = torch.zeros(source.shape[0], source.shape[1] + 1, dtype=source.dtype, device=source.device)
mask[(torch.arange(source.shape[0]), lengths)] = 1
mask = mask.cumsum(dim=1)[:, :-1] == 0

# expand the mask to dim=2 as well and pull the elements
out = source[mask[..., None].expand(-1, -1, source.shape[2])]

# since you pull different number of columns per row, you loose the shape of source. You need a final split to recover it
target = torch.split(out, (lengths * source.shape[2]).cpu().numpy().tolist())

target = [t_.view(-1, source.shape[2]) for t_ in target]

The output target is a list of 2-D tensors with varying number of rows (according to the counts of reverse_map and source.shape[2] columns in each.


Instruction: TypeError: expected CPU (got CUDA)

Input:
import torch
torch.cuda.is_available()
torch.cuda.current_device()
torch.cuda.get_device_name(0)
torch.cuda.memory_reserved()
torch.cuda.memory_allocated()
torch.cuda.memory_allocated()
var1=torch.FloatTensor([1.0,2.0,3.0]).cuda()
var1
var1.device
import pandas as pd
df=pd.read_csv('diabetes.csv')
df.head()
df.isnull().sum()

import seaborn as sns
import numpy as np
df['Outcome']=np.where(df['Outcome']==1,"Diabetic","No Diabetic")
df.head()
sns.pairplot(df,hue="Outcome")
X=df.drop('Outcome',axis=1).values### independent features
y=df['Outcome'].values###dependent features
from sklearn.model_selection import train_test_split

X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.2,random_state=0)
y_train
import torch
import torch.nn as nn
import torch.nn.functional as F
X_train=torch.FloatTensor(X_train).cuda()
X_test=torch.FloatTensor(X_test).cuda()
y_train=torch.LongTensor(y_train).cuda()
y_test=torch.LongTensor(y_test).cuda()

when I Run this code I got this error:
Traceback (most recent call last):
  File "<stdin>", line 24, in <module>
TypeError: expected CPU (got CUDA)

How to can I solve this error?

Output:
To transfer the variables to GPU, try the following:
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

X_train=torch.FloatTensor(X_train).to(device)
X_test=torch.FloatTensor(X_test).to(device)
y_train=torch.LongTensor(y_train).to(device)
y_test=torch.LongTensor(y_test).to(device)


Instruction: FFT loss in PyTorch

Input:
I want to compute the loss between the GT and the output of my network (called TDN) in the frequency domain by computing 2D FFT. The tensors are of dim batch x channel x height x width
amp_ip, phase_ip = 2DFFT(TDN(ip))
amp_gt, phase_gt = 2DFFT(TDN(gt))
loss = ||amp_ip - amp_gt||

For computing FFT I can use torch.fft(ip, signal_ndim = 2). But the output is in a + j b format i.e rectangular coordinates and NOT decomposed into phase and amplitude. How can I convert a + j b into amp exp(j phase) format in PyTorch? A side concern is also if signal_ndims be kept 2 to compute 2D FFT or something else?
The following description, which describes the loss that I plan to implement, maybe useful.

Output:
The question is answered by the GITHUB code file shared by @akshayk07 in the comments. Extracting the relevant information from that code, the concise answer to the question is,
    fft_im = torch.rfft(img.clone(), signal_ndim=2, onesided=False)

    # fft_im: size should be bx3xhxwx2
    fft_amp = fft_im[:,:,:,:,0]**2 + fft_im[:,:,:,:,1]**2
    fft_amp = torch.sqrt(fft_amp) # this is the amplitude
    fft_pha = torch.atan2( fft_im[:,:,:,:,1], fft_im[:,:,:,:,0] ) # this is the phase

As of PyTorch 1.7.1 choose torch.rfft over torch.fft as the latter does not work off the shelf with real valued tensors propagating in CNNs. Also a good idea will be ti use the normalisation flag of torch.rfft.


Instruction: PULSE on github (link provided) RuntimeError: CUDA out of memory.... preventing the program "run.py" from executing

Input:
(As a student I am kind of new to this but did quite a bit of research and I got pretty far, I'm super into learning something new through this!)
This issue is for the project pulse -> https://github.com/adamian98/pulse
the readme if you scroll down a bit on the page, gives a much better explanation than I could. It will also give a direct "correct" path to judge my actions against and make solving the problem a lot easier.
Objective: run program using the run.py file
Issue: I got a "RuntimeError: CUDA out of memory" despite having a compatible gpu and enough vram
Knowledge: when it comes to coding i just started a few days ago and have a dozen hours with anaconda now, comfterable creating environments.
What I did was... (the list below is a summary and the specific details are after it)

install anaconda

use this .yml file -> https://github.com/leihuayi/pulse/blob/feature/docker/pulse.yml (it changes dependencies to work for windows which is why I needed to grab a different one than the one supplied on the master github page) to create a new environment and install the required packages. It worked fantastically! I only got an error trying to install dlib, it didn't seem compatible with A LOT of the packages and my python version.

I installed the cuda toolkit 10.2 , cmake 3.17.2, and tried to install dlib into the environment directly. the errors spat out in a blaze of glory. The dlib package seems to be only needed for a different .py file and not run.py though so I think it may be unrelated to this error

logs are below and I explain my process in more detail
START DETAILS AND LOGS: from here until the "DETAILS 2" section should be enough information to solve, the rest past there is in case
error log for runing out of memory--> (after executing the "run.py" file)
Loading Synthesis Network
Loading Mapping Network
Running Mapping Network
Traceback (most recent call last):
File "C:\Users\micha\anaconda3\envs\Pulse1\pulse-master\run.py", line 58, in
model = PULSE(cache_dir=kwargs["cache_dir"])
File "C:\Users\micha\anaconda3\envs\Pulse1\pulse-master\PULSE.py", line 44, in init
latent_out = torch.nn.LeakyReLU(5)(mapping(latent))
File "C:\Users\micha\anaconda3\envs\pulse3\lib\site-packages\torch\nn\modules\module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "C:\Users\micha\anaconda3\envs\Pulse1\pulse-master\stylegan.py", line 233, in forward
x = super().forward(x)
File "C:\Users\micha\anaconda3\envs\pulse3\lib\site-packages\torch\nn\modules\container.py", line 100, in forward
input = module(input)
File "C:\Users\micha\anaconda3\envs\pulse3\lib\site-packages\torch\nn\modules\module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "C:\Users\micha\anaconda3\envs\Pulse1\pulse-master\stylegan.py", line 38, in forward
return F.linear(x, self.weight * self.w_mul, bias)
File "C:\Users\micha\anaconda3\envs\pulse3\lib\site-packages\torch\nn\functional.py", line 1610, in linear
ret = torch.addmm(bias, input, weight.t())
RuntimeError: CUDA out of memory. Tried to allocate 1.91 GiB (GPU 0; 6.00 GiB total capacity; 3.92 GiB already allocated; 744.91 MiB free; 3.93 GiB reserved in total by PyTorch)

End of error log.
NVIDIA-SMI LOG WHILE RUNNING (checking free memory)
C:\Users\micha>nvidia-smi --query-gpu=memory.free --format=csv --loop=1
memory.free [MiB]

5991 MiB
5991 MiB
5991 MiB
5991 MiB
5897 MiB
5781 MiB
5685 MiB
1643 MiB
5991 MiB
5991 MiB

the program stopped at the 1643MiB
DETAILS PART 1:
I have an nvidia gpu with 6gb of memory which, (according to other logs posted by the creator of the project-> should have enough memory for this to work "We ran our tests with 8GB of memory but I believe that you should be able to run the code with 4GB as well" -adamian98 ). I'm trying to fix that error and get the run.py to work as intended.
Here is the system info using the numba -s command in anaconda (includes hardware info: gpu, my windows version, memory, python version etc.)
START CUDA INFO

Hardware Information
Machine : AMD64
CPU Name : znver1
CPU Count : 16
Number of accessible CPUs : 16
List of accessible CPUs cores : 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
CFS Restrictions (CPUs worth of runtime) : None

CPU Features : 64bit adx aes avx avx2 bmi bmi2
clflushopt clwb clzero cmov cx16
cx8 f16c fma fsgsbase fxsr lzcnt
mmx movbe mwaitx pclmul popcnt
prfchw rdpid rdrnd rdseed sahf sha
sse sse2 sse3 sse4.1 sse4.2 sse4a
ssse3 wbnoinvd xsave xsavec
xsaveopt xsaves

Memory Total (MB) : 15789
Memory Available (MB) : 8421

OS Information
Platform Name : Windows-10-10.0.18362-SP0
Platform Release : 10
OS Name : Windows
OS Version : 10.0.18362
OS Specific Version : 10 10.0.18362 SP0 Multiprocessor Free
Libc Version : ?

Python Information
Python Compiler : MSC v.1916 64 bit (AMD64)
Python Implementation : CPython
Python Version : 3.8.5
Python Locale : en_CA.cp1252

LLVM Information
LLVM Version : 10.0.1

CUDA Information
CUDA Device Initialized : True
CUDA Driver Version : 10020
CUDA Detect Output:
Found 1 CUDA devices
id 0 b'GeForce GTX 1660 Ti with Max-Q Design' [SUPPORTED]
compute capability: 7.5
pci device id: 0
pci bus id: 1
Summary:
1/1 devices are supported

CUDA Librairies Test Output:
Finding cublas from
named cublas.dll
trying to open library... ERROR: failed to open cublas:
Could not find module 'cublas.dll' (or one of its dependencies). Try using the full path with constructor syntax.
Finding cusparse from
named cusparse.dll
trying to open library... ERROR: failed to open cusparse:
Could not find module 'cusparse.dll' (or one of its dependencies). Try using the full path with constructor syntax.
Finding cufft from
named cufft.dll
trying to open library... ERROR: failed to open cufft:
Could not find module 'cufft.dll' (or one of its dependencies). Try using the full path with constructor syntax.
Finding curand from
named curand.dll
trying to open library... ERROR: failed to open curand:
Could not find module 'curand.dll' (or one of its dependencies). Try using the full path with constructor syntax.
Finding nvvm from
named nvvm.dll
trying to open library... ERROR: failed to open nvvm:
Could not find module 'nvvm.dll' (or one of its dependencies). Try using the full path with constructor syntax.
Finding cudart from
named cudart.dll
trying to open library... ERROR: failed to open cudart:
Could not find module 'cudart.dll' (or one of its dependencies). Try using the full path with constructor syntax.
Finding libdevice from
searching for compute_20... ERROR: can't open libdevice for compute_20
searching for compute_30... ERROR: can't open libdevice for compute_30
searching for compute_35... ERROR: can't open libdevice for compute_35
searching for compute_50... ERROR: can't open libdevice for compute_50

ROC information
ROC Available : False
ROC Toolchains : None
HSA Agents Count : 0
HSA Agents:
None
HSA Discrete GPUs Count : 0
HSA Discrete GPUs : None

SVML Information
SVML State, config.USING_SVML : True
SVML Library Loaded : True
llvmlite Using SVML Patched LLVM : True
SVML Operational : True

Threading Layer Information
TBB Threading Layer Available : False
+--> Disabled due to Unknown import problem.
OpenMP Threading Layer Available : True
+-->Vendor: MS
Workqueue Threading Layer Available : True
+-->Workqueue imported successfully.

Numba Environment Variable Information
None found.

Conda Information
Conda Build : 3.20.5
Conda Env : 4.9.2
Conda Platform : win-64
Conda Python Version : 3.8.5.final.0
Conda Root Writable : True

Installed Packages
blas 1.0 mkl
ca-certificates 2020.12.8 haa95532_0
certifi 2020.12.5 py38haa95532_0
cffi 1.14.0 py38h7a1dbc1_0
chardet 3.0.4 py38haa95532_1003
cryptography 2.9.2 py38h7a1dbc1_0
cudatoolkit 10.2.89 h74a9793_1 anaconda
cycler 0.10.0 py38_0
freetype 2.9.1 ha9979f8_1
icc_rt 2019.0.0 h0cc432a_1
icu 58.2 ha925a31_3
idna 2.9 py_1
intel-openmp 2019.4 245
jpeg 9b hb83a4c4_2
kiwisolver 1.2.0 py38h74a9793_0
libcxx 7.0.0 h1ad3211_1002 conda-forge
libpng 1.6.37 h2a8f88b_0
libtiff 4.1.0 h56a325e_0
llvm-meta 7.0.0 0 conda-forge
m2-bash 4.3.042 5
m2-gcc-libs 5.3.0 4
m2-libedit 3.1 20150326
m2-libffi 3.2.1 2
m2-libreadline 6.3.008 8
m2-msys2-runtime 2.5.0.17080.65c939c 3
m2-ncurses 6.0.20160220 2
m2w64-gcc-libgfortran 5.3.0 6
m2w64-gcc-libs-core 5.3.0 7
m2w64-gmp 6.1.0 2
m2w64-libwinpthread-git 5.0.0.4634.697f757 2
matplotlib 3.1.3 py38_0
matplotlib-base 3.1.3 py38h64f37c6_0
mkl 2019.4 245
mkl-service 2.3.0 py38h196d8e1_0
mkl_fft 1.0.15 py38h14836fe_0
mkl_random 1.1.0 py38hf9181ef_0
msys2-conda-epoch 20160418 1
ninja 1.9.0 py38h74a9793_0
numpy 1.18.1 py38h93ca92e_0
numpy-base 1.18.1 py38hc3f5095_1
olefile 0.46 py_0
openssl 1.1.1i h2bbff1b_0
pandas 1.0.3 py38h47e9c7a_0
pillow 7.1.2 py38hcc1f983_0
pip 20.0.2 py38_3
powershell_shortcut 0.0.1 3
pycparser 2.20 py_2
pyopenssl 19.1.0 pyhd3eb1b0_1
pyparsing 2.4.7 py_0
pyqt 5.9.2 py38ha925a31_4
pysocks 1.7.1 py38haa95532_0
python 3.8.2 he1778fa_13
python-dateutil 2.8.1 py_0
pytorch 1.5.0 py3.8_cuda102_cudnn7_0 pytorch
pytz 2020.1 py_0
qt 5.9.7 vc14h73c81de_0
requests 2.23.0 py38_0
scipy 1.4.1 py38h9439919_0
setuptools 46.2.0 py38_0
sip 4.19.13 py38ha925a31_0
six 1.14.0 py38haa95532_0
sqlite 3.31.1 h2a8f88b_1
tk 8.6.8 hfa6e2cd_0
torchvision 0.6.0 py38_cu102 pytorch
tornado 6.0.4 py38he774522_1
urllib3 1.25.8 py38_0
vc 14.2 h21ff451_1
vs2015_runtime 14.27.29016 h5e58377_2
wheel 0.34.2 py38_0
win_inet_pton 1.1.0 py38haa95532_0
wincertstore 0.2 py38_0
xz 5.2.5 h62dcd97_0
zlib 1.2.11 h62dcd97_4
zstd 1.3.7 h508b16e_0

No errors reported.

Warning log
Warning (roc): Error initialising ROC: No ROC toolchains found.
Warning (roc): No HSA Agents found, encountered exception when searching: Error at driver init:

HSA is not currently supported on this platform (win32).`

END CUDA INFO
The section below is most likely not needed for this question, here in case:
DETAILS PART 2 (It sounded like dlib (and cmake) were only required to align the faces before therefore theoretically it isn't needed in run.py, but I'm not 100 percent sure. I omitted the section including the dlib specific errors):
I got the .yml installed successfully into the anaconda environment I named "pulse3" with no errors as well as the CUDA toolkit 10.2 and cmake 3.17.2. Dlib is the only thing that gave me some trouble spiting out a bunch of incompatibility errors with the other packages on windows with my python version 3.8.2.

Output:
based on new log evidence using this script simultaneously alongside the run.py file

 "C:\Users\micha>nvidia-smi --query-gpu=memory.free --format=csv --loop=1"

alongside using gpuz to monitor,
as well as the error produced from executing run.py in an anaconda environment with all required packages except dlib installed correctly:

RuntimeError: CUDA out of memory. Tried to allocate 1.91 GiB (GPU 0; 6.00 GiB total capacity; 3.92 GiB already allocated; 744.91 MiB free; 3.93 GiB reserved in total by PyTorch)

resulted in a logged usage of up to but not exceeding 4.502GB of VRAM on a gtx 1660ti
Therefore, I can  strongly infer that the project PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models (https://github.com/adamian98/pulse) does need somewhere between 6.01GB and 8.00GB of VRAM to execute the "run.py" file, therefore a gpu with 6gb is insufficient
so the error:

RuntimeError: CUDA out of memory. Tried to allocate 1.91 GiB (GPU 0; 6.00 GiB total capacity; 3.92 GiB already allocated; 744.91 MiB free; 3.93 GiB reserved in total by PyTorch)

is most likely due to hardware limitations.
Notes To Help Improve the Pulse Project:
This result, though inconclusive, contradicts this theory from the creator of the project:
"Unfortunately 2GB is not enough memory to store all of the gradients necessary during optimization. We ran our tests with 8GB of memory but I believe that you should be able to run the code with 4GB as well." -adamian98
Possible factors of error (see logs in question for more info if needed):
1.needing to execute the .py file from outside the anaconda environment (despite having the file inside the environment as well)
2.lack of successful dlib package installation
3.the input image not being the correct size (it was 75x77 pixels, and was yoinked from the project samples)
4.the input image not being the correct format (it was.png)

the image not being in the correct place \anaconda3\envs\pulse3\Library\qml\Qt3D
(the anaconda environment was named pulse3 that is why the path uses pulse3)


Instruction: How to load model for inference?

Input:
I am using a simple perceptron based classifier to generate sentiment analysis in Pytorch, complete code here Classifying Yelp Reviews.
The example does sentiment analysis, it outputs if the given input string is positive or negative.

example: this is a pretty small old great book -> positive

The application stores the final model along with the vectorizer.json.
So my question is: What should be the pre-requisites to build a separate application just to test the model, so that it can be used in web-application later-on?
Below is my current understanding and queries for this:
I assume to test, we need to load the model, load model parameters and evaluate for inference, please confirm

model = TheModelClass(*args, **kwargs) # Model class must be defined somewhere
model.load_state_dict(torch.load(PATH))
model.eval() # run if you only want to use it for inference

Once step 1 is done, I hope we can deploy the model using Flask and expose a REST API for model inference.

Output:
I found a nice tutorial explaining how to load model for inference, here is the link
https://pytorch.org/tutorials/recipes/recipes/saving_and_loading_models_for_inference.html
Thanks!


Instruction: nn.LSTM() received an invalid combination of arguments

Input:
I use lstm from pytorch in my code to predict time series. while i write this code
class LSTM_model(nn.Module):
    def __init__(self, input_size, output_size, hidden_size,num_layers,dropout):
        super(LSTM_model,self).__init__()

        self.input_size = input_size
        self.hidden_size = hidden_size
        self.seq_len = seq_len
        self.num_layers = num_layers
        self.dropout = dropout
        self.output_size = output_size

        # self.lstm=nn.LSTM(self.input_dim, self.hidden_dim, self.num_layers)
        self.lstm = nn.LSTM(self.input_size, self.hidden_size,self.num_layers , self.dropout, batch_first=True)
        self.fc= nn.Linear(self.hidden_size , self.output_size)

    def forward(self, x , hidden):
        x, hidden= self.lstm(x,hidden)
        x = self.fc(x)
        return x,hidden

but when I use the class, I get an error with the x,hidden=self.lstm(x,hidden) line about the internal nn.LSTM() function from PyTorch.
<ipython-input-63-211c1442b5a7> in forward(self, x, hidden)
     15
     16     def forward(self, x , hidden):
---> 17         x, hidden= self.lstm(x,hidden)
     18         x = self.fc(x)
     19         return x,hidden

D:\Anaconda\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

D:\Anaconda\lib\site-packages\torch\nn\modules\rnn.py in forward(self, input, hx)
    232         _impl = _rnn_impls[self.mode]
    233         if batch_sizes is None:
--> 234             result = _impl(input, hx, self._flat_weights, self.bias, self.num_layers,
    235                            self.dropout, self.training, self.bidirectional, self.batch_first)
    236         else:

TypeError: rnn_tanh() received an invalid combination of arguments - got (Tensor, Tensor, list, int, int, float, bool, bool, bool), but expected one of:
 * (Tensor data, Tensor batch_sizes, Tensor hx, tuple of Tensors params, bool has_biases, int num_layers, float dropout, bool train, bool bidirectional)
      didn't match because some of the arguments have invalid types: (Tensor, Tensor, !list!, !int!, !int!, !float!, !bool!, bool, bool)
 * (Tensor input, Tensor hx, tuple of Tensors params, bool has_biases, int num_layers, float dropout, bool train, bool bidirectional, bool batch_first)
      didn't match because some of the arguments have invalid types: (Tensor, Tensor, !list!, !int!, int, float, bool, bool, bool)

i called the function with this line
model = LSTM_model(input_size=1, output_size=1, hidden_size=128, num_layers=2, dropout=0).to(device)

and its called here
from tqdm.auto import tqdm
def loop_fn(mode, dataset, dataloader, model, criterion, optimizer,device):
    if mode =="train":
        model.train()
    elif mode =="test":
        model.eval()
    cost = 0
    for feature, target in tqdm(dataloader, desc=mode.title()):
        feature, target = feature.to(device), target.to(device)
        output , hidden = model(feature,None)
        loss = criterion(output,target)

        if mode =="train":
            loss.backward()
            optimizer.step()
            optimizer.zero_grad()

        cost += loss.item() * feature.shape[0]
    cost = cost / len(dataset)
    return cost

Thank You in advance

Output:
I took me a while to find out, but you are initializing your nn.LSTM incorrectly because of positional arguments.
self.lstm = nn.LSTM(self.input_size, self.hidden_size,
    self.num_layers, self.dropout, batch_first=True)

The above will assign self.dropout to the argument named bias:
>>> model.lstm
LSTM(1, 128, num_layers=2, bias=0, batch_first=True)

You may want to use keyword arguments instead:
self.lstm = nn.LSTM(
    input_size=self.input_size,
    hidden_size=self.hidden_size,
    num_layers=self.num_layers,
    dropout=self.dropout,
    batch_first=True)

Which will provide the desired result:
>>> model.lstm
LSTM(1, 128, num_layers=2, batch_first=True)


Instruction: Output of CNN should be image

Input:
I am pretty new to deep learning, so I got one question:
Assume an input Grayscale image of shape (128,128,1). Target (Output) is as well an (128,128,1) sized image, e.g. for segmentation, depth prediction etc.. Usually with valid padding the size of the image shrinks after several convolution layers.
What are decent (maybe not the toughest one) variants to keep the size or predict a same sized image? Is it via same-padding? Is it via tranpose convolution or upsampling? Should I use a FCN at the end and reshape them to the image size? I am using pytorch. I would be glad for any hints, because I didn't find much in the internet.
Best

Output:
TLDR; You want to look at Deconv networks (Convolution transpose) that help regenerate an image using convolution operations. You want to build an encoder-decoder convolution architecture that compresses an image to a latent representation using convolutions and then decodes an image from this compressed representation. For image segmentation, a popular architecture is U-net.

NOTE: I cant answer for pytorch, so I will he sharing the Tensorflow equivalent. Please feel to ignore the code, but since you are looking for the concept, I can help you with what you need to solve this.
You are trying to generate an image as the output of the network.
A series convolution operation help to Downsample an image. Since you need an output 2D matrix (gray scale image), you want to Upsample as well. Such a network is called a Deconv network.
The first series of layers convolve over the input, 'flattening' them into a vector of channels. The next set of layers use 2D Conv Transpose or Deconv operations to change the channels back into a 2D matrix (Gray scale image)
Refer to this image for reference -

Here is a sample code that shows you how you can take a (10,3,1) image to a (12,10,1) image using a deconv net.

You can find the conv2dtranspose layer implementation in pytorch here.

from tensorflow.keras import layers, Model, utils

inp = layers.Input((128,128,1))  ##
x = layers.Conv2D(2, (3,3))(inp) ##  Convolution part
x = layers.Conv2D(4, (3,3))(x)   ##
x = layers.Conv2D(6, (3,3))(x)   ##

##########

x = layers.Conv2DTranspose(6, (3,3))(x)
x = layers.Conv2DTranspose(4, (3,3))(x)   ##   ##  Deconvolution part
out = layers.Conv2DTranspose(1, (3,3))(x) ##

model = Model(inp, out)
utils.plot_model(model, show_shapes=True, show_layer_names=False)

Also, if you are looking for tried and tested architectures in this domain, check out U-net; U-Net: Convolutional Networks for Biomedical Image Segmentation. This is an encoder-decoder (conv2d, conv2d-transpose) architecture that uses a concept called skip connections to avoid information loss and generate better image segmentation masks.


Instruction: Problem with adding smiles on photos with convolutional autoencoder

Input:
I have a dataset with images and another dataset as it's description:

There are a lot of pictures: people with and without sunglasses, smiles and other attributes. What I want to do is be able to add smiles to photos where people are not smiling.
I've started like this:
smile_ids = attrs['Smiling'].sort_values(ascending=False).iloc[100:125].index.values
smile_data = data[smile_ids]

no_smile_ids = attrs['Smiling'].sort_values(ascending=True).head(5).index.values
no_smile_data = data[no_smile_ids]

eyeglasses_ids = attrs['Eyeglasses'].sort_values(ascending=False).head(25).index.values
eyeglasses_data = data[eyeglasses_ids]

sunglasses_ids = attrs['Sunglasses'].sort_values(ascending=False).head(5).index.values
sunglasses_data = data[sunglasses_ids]

When I print them their are fine:
plot_gallery(smile_data, IMAGE_H, IMAGE_W, n_row=5, n_col=5, with_title=True, titles=smile_ids)

Plot gallery looks like this:
def plot_gallery(images, h, w, n_row=3, n_col=6, with_title=False, titles=[]):
plt.figure(figsize=(1.5 * n_col, 1.7 * n_row))
plt.subplots_adjust(bottom=0, left=.01, right=.99, top=.90, hspace=.35)
for i in range(n_row * n_col):
    plt.subplot(n_row, n_col, i + 1)
    try:
        plt.imshow(images[i].reshape((h, w, 3)), cmap=plt.cm.gray, vmin=-1, vmax=1, interpolation='nearest')
        if with_title:
            plt.title(titles[i])
        plt.xticks(())
        plt.yticks(())
    except:
        pass

Then I do:
def to_latent(pic):
with torch.no_grad():
    inputs = torch.FloatTensor(pic.reshape(-1, 45*45*3))
    inputs = inputs.to('cpu')
    autoencoder.eval()
    output = autoencoder.encode(inputs)
    return output

def from_latent(vec):
with torch.no_grad():
    inputs = vec.to('cpu')
    autoencoder.eval()
    output = autoencoder.decode(inputs)
    return output

After that:
smile_latent = to_latent(smile_data).mean(axis=0)
no_smile_latent = to_latent(no_smile_data).mean(axis=0)
sunglasses_latent = to_latent(sunglasses_data).mean(axis=0)

smile_vec = smile_latent-no_smile_latent
sunglasses_vec = sunglasses_latent - smile_latent

And finally:
def add_smile(ids):
for id in ids:
    pic = data[id:id+1]
    latent_vec = to_latent(pic)
    latent_vec[0] += smile_vec
    pic_output = from_latent(latent_vec)
    pic_output = pic_output.view(-1,45,45,3).cpu()
    plot_gallery([pic,pic_output], IMAGE_H, IMAGE_W, n_row=1, n_col=2)

def add_sunglasses(ids):
for id in ids:
    pic = data[id:id+1]
    latent_vec = to_latent(pic)
    latent_vec[0] += sunglasses_vec
    pic_output = from_latent(latent_vec)
    pic_output = pic_output.view(-1,45,45,3).cpu()
    plot_gallery([pic,pic_output], IMAGE_H, IMAGE_W, n_row=1, n_col=2)

But when I execute this line I don't get any faces:
add_smile(no_smile_ids)

The output:

Could someone please explain where is my mistake or why it can happen? Thanks for any help.
Added: checking the shape of pic_output:

Output:
Wild guess, but it seems you are broadcasting your images instead of permuting the axes. The former will have the undesired effect of mixing information across the batches/channels.
pic_output = pic_output.view(-1, 45, 45, 3).cpu()

should be replaced with
pic_output = pic_output.permute(0, 2, 3, 1).cpu()

Assuming tensor pic_output is already shaped like (-1, 3, 45, 45).


Instruction: How can I make this PyTorch heatmap function faster and more efficient?

Input:
I have this function that creates a sort if heatmap for 2d tensors, but it's painfully slow when using larger tensor inputs. How can I speed it up and make it more efficient?
import torch
import numpy as np
import matplotlib.pyplot as plt

def heatmap(
    tensor: torch.Tensor,
) -> torch.Tensor:
    assert tensor.dim() == 2

    def color_tensor(x: torch.Tensor) -> torch.Tensor:
        if x < 0:
            x = -x
            if x < 0.5:
                x = x * 2
                return (1 - x) * torch.tensor(
                    [0.9686, 0.9686, 0.9686]
                ) + x * torch.tensor([0.5725, 0.7725, 0.8706])
            else:
                x = (x - 0.5) * 2
                return (1 - x) * torch.tensor(
                    [0.5725, 0.7725, 0.8706]
                ) + x * torch.tensor([0.0196, 0.4431, 0.6902])
        else:
            if x < 0.5:
                x = x * 2
                return (1 - x) * torch.tensor(
                    [0.9686, 0.9686, 0.9686]
                ) + x * torch.tensor([0.9569, 0.6471, 0.5098])
            else:
                x = (x - 0.5) * 2
                return (1 - x) * torch.tensor(
                    [0.9569, 0.6471, 0.5098]
                ) + x * torch.tensor([0.7922, 0.0000, 0.1255])

    return torch.stack(
        [torch.stack([color_tensor(x) for x in t]) for t in tensor]
    ).permute(2, 0, 1)

x = torch.randn(3,3)
x = x / x.max()
x_out = heatmap(x)

x_out = (x_out.permute(1, 2, 0) * 255).numpy()
plt.imshow(x_out.astype(np.uint8))
plt.axis("off")
plt.show()

An example of the output:

Output:
You need to get rid of ifs and the for loop and make a vectorized function. To do that, you can use masks and calculate all in one. Here it is:

def heatmap(tensor: torch.Tensor) -> torch.Tensor:
    assert tensor.dim() == 2

    # We're expanding to create one more dimension, for mult. to work.
    xt = x.expand((3, x.shape[0], x.shape[1])).permute(1, 2, 0)

    # this part is the mask: (xt >= 0) * (xt < 0.5) ...
    # ... the rest is the original function translated
    color_tensor = (
        (xt >= 0) * (xt < 0.5) * ((1 - xt * 2) * torch.tensor([0.9686, 0.9686, 0.9686]) + xt * 2 * torch.tensor([0.9569, 0.6471, 0.5098]))
        +
        (xt >= 0) * (xt >= 0.5) * ((1 - (xt - 0.5) * 2) * torch.tensor([0.9569, 0.6471, 0.5098]) + (xt - 0.5) * 2 * torch.tensor([0.7922, 0.0000, 0.1255]))
        +
        (xt < 0) * (xt > -0.5) * ((1 - (-xt * 2)) * torch.tensor([0.9686, 0.9686, 0.9686]) + (-xt * 2) * torch.tensor([0.5725, 0.7725, 0.8706]))
        +
        (xt < 0) * (xt <= -0.5) * ((1 - (-xt - 0.5) * 2) * torch.tensor([0.5725, 0.7725, 0.8706]) + (-xt - 0.5) * 2 * torch.tensor([0.0196, 0.4431, 0.6902]))
    ).permute(2, 0, 1)

    return color_tensor


Instruction: How to use VGG19 transfer learning pretraining

Input:
I'm working on a VQA model, and I need some help as I'm new to this.
I want to use transfer learning from the VGG19 network before running the train, so when I start the train, I will have the image features ahead (trying to solve performance issue).
Does it possible to do so? If so, can someone please share an example with pytorch?
below is the relevant code:
class img_CNN(nn.Module):
  def __init__(self, img_size):

        super(img_CNN, self).__init__()
        self.model = models.vgg19(pretrained=True)
        self.in_features = self.model.classifier[-1].in_features
        self.model.classifier = nn.Sequential(*list(self.model.classifier.children())[:-1]) # remove vgg19 last layer
        self.fc = nn.Linear(in_features, img_size)

  def forward(self, image):
    #with torch.no_grad():
    img_feature = self.model(image) # (batch, channel, height, width)
    img_feature = self.fc(img_feature)
    return img_feature

class vqamodel(nn.Module):
  def __init__(self, output_dim,input_dim, emb_dim, hid_dim, n_layers, dropout, answer_len, que_size, img_size,model_vgg,in_features):
    super(vqamodel,self).__init__()
    self.image=img_CNN(img_size)
    self.question=question_lstm(input_dim, emb_dim, hid_dim, n_layers, dropout,output_dim,que_size)
    self.tanh=nn.Tanh()
    self.relu=nn.ReLU()
    self.dropout=nn.Dropout(dropout)
    self.fc1=nn.Linear(que_size,answer_len) #the input to the linear network is equal to the combain vector
    self.softmax=nn.Softmax(dim=1)

  def forward(self, image, question):
    image_emb=self.image(image)
    question_emb=self.question(question)
    combine =question_emb*image_emb
    out_feature=self.fc1(combine)
    out_feature=self.relu(out_feature)

    return (out_feature)

How can I take out the models.vgg19(pretrained=True),run it before the train on the image dataloader and save the image representation in NumPy array?
thank you!

Output:
Yes, you can use a pretrained VGG model to extract embedding vectors from images. Here is a possible implementation, using torchvision.models.vgg*.

First retrieve the pretrained model
model = torchvision.models.vgg19(pretrained=True)

Its classifier is:
>>> model.classifier
(classifier): Sequential(
    (0): Linear(in_features=25088, out_features=4096, bias=True)
    (1): ReLU(inplace=True)
    (2): Dropout(p=0.5, inplace=False)
    (3): Linear(in_features=4096, out_features=4096, bias=True)
    (4): ReLU(inplace=True)
    (5): Dropout(p=0.5, inplace=False)
    (6): Linear(in_features=4096, out_features=1000, bias=True)
)

Depending on your finetuning strategy, you can either truncate it to keep some of the  trained dense layers:
model.classifier = nn.Sequential(*[model.classifier[i] for i in range(4)])

Or replace it altogether with a different set of dense layers wrapped in a nn.Sequential:
model.classifier = nn.Sequential(
    nn.Linear(25088, 4096),
    nn.ReLU(True),
    nn.Dropout(0.5),
    nn.Linear(4096, 2048))

Additionally, you can freeze the entire head of the model (the feature extractor):
for param in model.features.parameters():
    param.requires_grad = False

Then you will be able to use that model to extract image embeddings and perform back propagation to finetune your classifier:
>>> model(img) # shape (batchs_size, 2048)


Instruction: Why is the code not able to approximate the square function?

Input:
WHy does the following code not work as a square approximator? I am getting weird dimensions. When I tried plotting loss, the graph somehow does not show anything. I am a beginner with pytorch, so I would be grateful for any help.
import torch
from torch import nn
import matplotlib.pyplot as plt
import numpy as np

data = [[i] for i in range(-10000, 10000)]
y = [[i[0] * i[0]] for i in data]
data=torch.FloatTensor(data)
y=torch.FloatTensor(y)

class MyModel(nn.Module):
    def __init__(self, numfeatures, outfeatures):
        super().__init__()
        self.modele = nn.Sequential(
                                    nn.Linear( numfeatures,  2*numfeatures),
                                    nn.ReLU(),
                                    nn.Linear(2 * numfeatures, 4 * numfeatures),
                                    nn.ReLU(),
                                    nn.Linear(4* numfeatures, 2 * numfeatures),
                                    nn.ReLU(),
                                    nn.Linear(2*numfeatures, numfeatures),
                                    )
    def forward(self, x):
        return self.modele(x)

model = MyModel(1, 1)
criterion = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

n_epochs = 10000
epoch_loss= []

for i in range(n_epochs):
    y_pred = model(data)
    loss = criterion(y_pred, y)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    epoch_loss.append(loss.item())

plt.plot(epoch_loss)

Output:
Your data is ranging from -10000 to 10000! You need to standardize your data, otherwise you won't be able to make your model learn:
data = (data - data.min()) / (data.max() - data.min())
y = (y - y.min()) / (y.max() - y.min())

Additionally, you could normalize your input with:
mean, std = data.mean(), data.std()
data = (data - mean) / std

After 100 epochs:


Instruction: when I run this code in Pycharm i didn't get any outputs simply it shows------ process finished with exit code 0. please help me with this issue

Input:
import torch
torch.cuda.is_available()
torch.cuda.current_device()
torch.cuda.get_device_name(0)
torch.cuda.memory_reserved()
torch.cuda.memory_allocated()
torch.cuda.memory_allocated()
var1=torch.FloatTensor([1.0,2.0,3.0]).cuda()
var1
var1.device
import pandas as pd
df=pd.read_csv('diabetes.csv')
df.head()
df.isnull().sum()
import seaborn as sns
import numpy as np
df['Outcome']=np.where(df['Outcome']==1,"Diabetic","No Diabetic")
df.head()
sns.pairplot(df,hue="Outcome")

The output shows:
C:\Users\vinot.conda\envs\python21\python.exe D:/python/python_work/pythonProject/diabetes.py
Process finished with exit code 0

Output:
You have to print your outputs, for example:
torch.cuda.is_available()

Will return either True/False but it isn't assigned to any variable and hence it does nothing. It doesn't work like Jupyter notebooks.
You can assign your outputs to variable, just like you did with
df=pd.read_csv('diabetes.csv')

Or you can print dataframe (or any other variable that is not assigned for that matter, not sure what you're after):
print(df.head())


Instruction: Which parameters of Mask-RCNN control mask recall?

Input:
I'm interested in fine-tuning a Mask-RCNN model that I'm using for instance segmentation. Currently I have trained the model for 6 epochs and the various Mask-RCNN losses are as follows:

The reason I'm stopping is that the COCO evaluation metrics seem to have dipped in the last epoch:

I know this is a far reaching question, but I'm looking to gain some intuition of how to understand which parameters are going to be the most impactful in improving the evaluation metrics. I understand there are three places to consider:

Should I be looking at batch size, learning rate and momentum, this uses an SGD optimizer with a learning rate of 1e-4 and batch size 2?
Should I be looking at using more training data or adding augmentation (I don't currently use any) and my dataset is current pretty large 40K images?
Should I be looking at the specific MaskRCNN parameters?

I thing I'll likely be asked to me more specific on what I want to improve so let me say that I would like to improve the recall of the individual masks. The model is performing well but doesn't quite capture the full extend of what I would like it to. I'm also leaving out details of the specific learning problem as I'd like to gain intuition of how to approach this in general.

Output:
A couple of notes:

6 epochs is a way too little number for the network to converge even if you use a pre-trained network. Especially such a big one as resnet50. I think you need at least 50 epochs. On a pre-trained resnet18 I started to get good results after 30 epochs, resnet34 needed +10-20 epochs and your resnet50 + 40k images of train set - definitely need more epochs than 6;
definitely use a pre-trained network;
in my experience, I failed to get results I like with SGD. I started using AdamW + ReduceLROnPlateau scheduler. Network converges quite fast, like 50-60% AP on epoch 7-8 but then it comes up to 80-85 after 50-60 epochs using very small improvements from epoch to epoch, only if the LR is small enough. You must be familiar with the gradient descent notion. I used to think of it as if you have more augmentation, your "hill" is covered with "boulders" that you have to be able to bypass and this is only possible if you control the LR. Additionally, AdamW helps with the overfitting.

This is how I do it. For networks with higher input resolution (you input images are scaled on input by the net itself), I use higher lr.
init_lr = 0.00005
weight_decay = init_lr * 100
optimizer = torch.optim.AdamW(params, lr=init_lr, weight_decay=weight_decay)
scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, verbose=True, patience=3, factor=0.75)

for epoch in range(epochs):
    # train for one epoch, printing every 10 iterations
    metric_logger = train_one_epoch(model, optimizer, train_loader, scaler, device,
                                    epoch, print_freq=10)

    scheduler.step(metric_logger.loss.global_avg)
    optimizer.param_groups[0]["weight_decay"] = optimizer.param_groups[0]["lr"] * 100

    # scheduler.step()

    # evaluate on the test dataset
    evaluate(model, test_loader, device=device)

    print("[INFO] serializing model to '{}' ...".format(args["model"]))
    save_and_print_size_of_model(model, args["model"], script=False)

Find such an lr and weight decay that the training exhausts lr to a very small value, like 1/10 of your initial lr, at the end of the training. If you will have a plateau too often, the scheduler quickly brings it to very small values and the network will learn nothing all the rest of the epochs.
Your plots indicate that your LR is too high at some point of the training, the network stops training and then AP is going down. You need constant improvements, even small ones. The more network trains the more subtle details it learns about your domain and the smaller is the learning rate. Imho, constant LR will not allow doing that correctly.

anchor generator settings. Here is how I initialize the network.
 def get_maskrcnn_resnet_model(name, num_classes, pretrained, res='normal'):
      print('Using maskrcnn with {} backbone...'.format(name))

      backbone = resnet_fpn_backbone(name, pretrained=pretrained, trainable_layers=5)

      sizes = ((4,), (8,), (16,), (32,), (64,))
      aspect_ratios = ((0.25, 0.5, 1.0, 2.0, 4.0),) * len(sizes)
      anchor_generator = AnchorGenerator(
          sizes=sizes, aspect_ratios=aspect_ratios
      )

      roi_pooler = torchvision.ops.MultiScaleRoIAlign(featmap_names=['0', '1', '2', '3'],
                                                      output_size=7, sampling_ratio=2)

      default_min_size = 800
      default_max_size = 1333

      if res == 'low':
          min_size = int(default_min_size / 1.25)
          max_size = int(default_max_size / 1.25)
      elif res == 'normal':
          min_size = default_min_size
          max_size = default_max_size
      elif res == 'high':
          min_size = int(default_min_size * 1.25)
          max_size = int(default_max_size * 1.25)
      else:
          raise ValueError('Invalid res={} param'.format(res))

      model = MaskRCNN(backbone, min_size=min_size, max_size=max_size, num_classes=num_classes,
                       rpn_anchor_generator=anchor_generator, box_roi_pool=roi_pooler)

      model.roi_heads.detections_per_img = 512
      return model

I need to find small objects here why I use such anchor params.

classes in-balancing issue. If you have only your object and bg - no problem. If you have more classes then make sure that your training split (as 80% for train and 20% for the test) is more or less precisely applied to all the classes used in your particular training.

Good luck!


Instruction: Pytorch in V.S. Code: torch.tensor is not callable

Input:
I am running my pytorch program in V.S. Code, but, I am getting the following errors:-
torch.tensor is not callable

I have followed this answer https://stackoverflow.com/a/53572939/13218692 and added the following lines in my Settings.json file in V.S. Code, but, I am still getting an error. Any help would be highly appreciated! Thanks!

Output:
add this line after it: ,"--disable-msg=not-callable"
just like this enter image description here


Instruction: How to run Python script on a Discrete Graphics AMD GPU?

Input:
WHAT I WANT TO DO:

I have a script that I use for factorizing prime numbers given a certain range:
# Python program to display all the prime numbers within an interval

lower = 900
upper = 1000

print("Prime numbers between", lower, "and", upper, "are:")

for num in range(lower, upper + 1):
   # all prime numbers are greater than 1
   if num > 1:
       for i in range(2, num):
           if (num % i) == 0:
               break
       else:
           print(num)

I would like to use the GPU instead of the CPU to run such script so it would be faster

THE PROBLEM:

I don't have a NVIDIA GPU on my Intel NUC NUC8i7HVK but a "Discrete GPU"

If I run this code to check what are my GPUs:
import pyopencl as cl
import numpy as np

a = np.arange(32).astype(np.float32)
res = np.empty_like(a)

ctx = cl.create_some_context()
queue = cl.CommandQueue(ctx)

mf = cl.mem_flags
a_buf = cl.Buffer(ctx, mf.READ_ONLY | mf.COPY_HOST_PTR, hostbuf=a)
dest_buf = cl.Buffer(ctx, mf.WRITE_ONLY, res.nbytes)

prg = cl.Program(ctx, """
    __kernel void sq(__global const float *a,
    __global float *c)
    {
      int gid = get_global_id(0);
      c[gid] = a[gid] * a[gid];
    }
    """).build()

prg.sq(queue, a.shape, None, a_buf, dest_buf)

cl.enqueue_copy(queue, res, dest_buf)

print (a, res)

I receive:

[0] <pyopencl.Platform 'AMD Accelerated Parallel Processing' at 0x7ffb3d492fd0>
[1] <pyopencl.Platform 'Intel(R) OpenCL HD Graphics' at 0x187b648ed80>

THE POSSIBLE APPROACH TO THE PROBLEM:

I found a guide that takes you by the hand and explains step by step how to run it on your GPU. But all Pyhton libraries that pipes Python through the GPU like PyOpenGL, PyOpenCL, Tensorflow (Force python script on GPU), PyTorch, etc... are tailored for NVIDIA.
In case you have an AMD all libraries ask for ROCm but such software still doesn't support integrated GPU or Discrete GPU as far as I know (see my own reply below).
I only found a guide that talks about such approach but I cannot make it work.
Is there hope or I'm just tying to do something impossible?

EDIT: Reply to @chapelo

If I choose 0 the reply is:
Set the environment variable PYOPENCL_CTX='0' to avoid being asked again.
[ 0.  1.  2.  3.  4.  5.  6.  7.  8.  9. 10. 11. 12. 13. 14. 15. 16. 17.
 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31.] [  0.   1.   4.   9.  16.  25.  36.  49.  64.  81. 100. 121. 144. 169.
 196. 225. 256. 289. 324. 361. 400. 441. 484. 529. 576. 625. 676. 729.
 784. 841. 900. 961.]

If I choose 1 the reply is:
Set the environment variable PYOPENCL_CTX='1' to avoid being asked again.
[ 0.  1.  2.  3.  4.  5.  6.  7.  8.  9. 10. 11. 12. 13. 14. 15. 16. 17.
 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31.] [  0.   1.   4.   9.  16.  25.  36.  49.  64.  81. 100. 121. 144. 169.
 196. 225. 256. 289. 324. 361. 400. 441. 484. 529. 576. 625. 676. 729.
 784. 841. 900. 961.]

Output:
The following code is sample of a complete python program that usually includes:

The import statements
The function definitions
The main() function
The if __name__ == "__main__": section.

I hope this helps you solve your problem.
import pyprimes
from math import sqrt
import numpy as np

import pyopencl as cl
import pyopencl.algorithm
import pyopencl.array

def primes_below(number):
    """Generate a list of prime numbers below a specified  `number`"""
    n = lambda a: 2 if a==0 else 2*a + 1
    limit = int(sqrt(number)) + 1
    size = number//2
    primes = [True] * size
    for i in range(1, size):
        if primes[i]:
            num = n(i)
            if num > limit: break
            for j in range(i+num, size, num):
                primes[j] = False
    for i, flag in enumerate(primes):
        if flag:
            yield n(i)

def primes_between(lo, hi):
    """Generate a list of prime numbers betwenn `lo` and `hi` numbers"""
    primes = list(primes_below(int(sqrt(hi))+1))
    size = (hi - lo - (0 if hi%2 else 1))//2 + 1
    n = lambda a: 2*a + lo + (0 if lo%2 else 1)
    numbers = [True]*size
    for i, prime in enumerate(primes):
        if i == 0: continue # avoid dividing by 2
        nlo = n(0)
        # slower # start = prime * (nlo//prime + 1) if nlo%prime else 0
        start = 0
        while (n(start)%prime) != 0:
            start += 1
        for j in range(start, size, prime):
            numbers[j] = False
    for i, flag in enumerate(numbers):
        if flag:
            yield n(i)

def primes_between_using_cl(lo, hi):
    """Generate a list of prime numbers betwenn a lo and hi numbers
    this is a parallel algorithm using pyopencl"""
    primes = list(primes_below(int(sqrt(hi))+1))
    size_primes_h = np.array( (len(primes)-1, ), dtype=np.int32)
    numbers_h = np.arange(  lo + (0 if lo&1 else 1),
                                  hi + (0 if hi&1 else 1),
                                  2,
                                  dtype=np.int32)
    size = (hi - lo - (0 if hi%2 else 1))//2 + 1
    code = """\
    __kernel
    void is_prime( __global const int *primes,
                        __global         int *numbers) {
      int gid = get_global_id(0);
      int num = numbers[gid];
      int max = (int) (sqrt((float)num) + 1.0);
      for (; *primes; ++primes) {
         if (*primes > max) break;
         if (num % *primes == 0) {
            numbers[gid] = 0;
            return;
         }
      }
    }
    """
    platforms = cl.get_platforms()
    ctx = cl.Context(dev_type=cl.device_type.ALL,
        properties=[(cl.context_properties.PLATFORM, platforms[0])])
    queue = cl.CommandQueue(ctx)
    prg = cl.Program(ctx, code).build()
    numbers_d = cl.array.to_device(queue, numbers_h)
    primes_d = cl.array.to_device(queue, np.array(primes[1:], dtype=np.int32))
    prg.is_prime(queue, (size, ), None, primes_d.data, numbers_d.data)
    array, length = cl.algorithm.copy_if(numbers_d, "ary[i]>0")[:2]
    yield from array.get()[:length.get()]

def test(f, lo, hi):
    """Test that all prime numbers are generated by comparing with the
    output of the library `pyprimes`"""
    a = filter(lambda p: p>lo, pyprimes.primes_below(hi))
    b = f(lo, hi)
    result = True
    for p, q in zip (a, b):
        if p != q:
            print(p, q)
            result = False
    return result

def main():
    lower = 1000
    upper = 5000
    print("The prime numbers between {} and {}, are:".format(lower,upper))
    print()
    for p in primes_between_using_cl(lower, upper):
        print(p, end=' ')
    print()

if __name__ == '__main__':
    main()


Instruction: PyTorch DataLoader using Mongo DB

Input:
I would like to know if using a DataLoader connected to a MongoDB is a sensible thing to do and how this could be implemented.
Background
I have about 20 million documents sitting in a (local) MongoDB. Way more documents than fit in memory. I would like to train a deep neural net on the data. So far, I have been exporting the data to the file system first, with subfolders named as the classes of the documents. But I find this approach nonsensical. Why export first (and later delete) if the data is already well-maintained sitting in a DB.
Question 1:
Am I right? Would it make sense to directly connect to the MongoDB? Or are there reasons not to do it (e.g. DBs generally being too slow etc.)? If DBs are too slow (why?), can one prefetch the data somehow?
Question 2:
How would one implement a PyTorch DataLoader?
I have found only very few code snippets online ([1] and [2]) which makes me doubt my approach.
Code snippet
The general way how I access MongoDB is as follows below. Nothing special about this, I think.
import pymongo
from pymongo import MongoClient

myclient = pymongo.MongoClient("mongodb://localhost:27017/")
mydb = myclient["xyz"]
mycol = mydb["xyz_documents"]

query = {
    # some filters
}

results = mycol.find(query)

# results is now a cursor that can run through all docs
# Assume, for the sake of this example, that each doc contains a class name and some image that I want to train a classifier on

Output:
Introduction
This one is a little open-ended, but let's try, also please correct me if I'm wrong somewhere.

So far, I have been exporting the data to the file system first, with subfolders named as the classes of the documents.

IMO this isn't sensible because:

you are essentially duplicating data
any time you would like to train a-new given only code and database this operation would have to be repeated
you can access multiple datapoints at once and cache them in RAM for later reuse without reading from hard drive multiple times (which is quite heavy)

Am I right? Would it make sense to directly connect to the MongoDB?

Given above, probably yes (especially when it comes to clear and portable implementation)

Or are there reasons not to do it (e.g. DBs generally being to slow etc.)?

AFAIK DB shouldn't be slower in this case as it will cache access to it, but I'm no db expert unfortunately. Many tricks for faster access are implemented out-of-the-box for databases.

can one prefetch the data somehow?

Yes, if you just want to get data, you could load a larger part of data (say 1024 records) at one go and return batches of data from that (say batch_size=128)
Implementation

How would one implement a PyTorch DataLoader? I have found only very few code snippets online ([1] and [2]) which makes me doubt with my approach.

I'm not sure why would you want to do that. What you should go for is torch.utils.data.Dataset as shown in the examples you've listed.
I would start with simple non-optimized approach similar to the one here, so:

open connection to db in __init__ and keep it as long as it's used (I would create a context manager from torch.utils.data.Dataset so the connection is closed after epochs are finished)
I would not transform the results to list (especially since you cannot fit it in RAM for obvious reasons) as it misses the point of generators
I would perform batching inside this Dataset (there is an argument batch_size here).
I am not sure about __getitem__ function but it seems it can return multiple datapoints at once, hence I'd use that and it should allow us to use num_workers>0 (given that mycol.find(query) returns data in the same order every time)

Given that, something along those lines is what I'd do:
class DatabaseDataset(torch.utils.data.Dataset):
    def __init__(self, query, batch_size, path: str, database: str):
        self.batch_size = batch_size

        client = pymongo.MongoClient(path)
        self.db = client[database]
        self.query = query
        # Or non-approximate method, if the approximate method
        # returns smaller number of items you should be fine
        self.length = self.db.estimated_document_count()

        self.cursor = None

    def __enter__(self):
        # Ensure that this find returns the same order of query every time
        # If not, you might get duplicated data
        # It is rather unlikely (depending on batch size), shouldn't be a problem
        # for 20 million samples anyway
        self.cursor = self.db.find(self.query)
        return self

    def shuffle(self):
        # Find a way to shuffle data so it is returned in different order
        # If that happens out of the box you might be fine without it actually
        pass

    def __exit__(self, *_, **__):
        # Or anything else how to close the connection
        self.cursor.close()

    def __len__(self):
        return len(self.examples)

    def __getitem__(self, index):
        # Read takes long, hence if you can load a batch of documents it should speed things up
        examples = self.cursor[index * batch_size : (index + 1) * batch_size]
        # Do something with this data
        ...
        # Return the whole batch
        return data, labels

Now batching is taken care of by DatabaseDataset, hence torch.utils.data.DataLoader can have batch_size=1. You might need to squeeze additional dimension.
As MongoDB uses locks (which is no surprise, but see here) num_workers>0 shouldn't be a problem.
Possible usage (schematically):
with DatabaseDataset(...) as e:
    dataloader = torch.utils.data.DataLoader(e, batch_size=1)
    for epoch in epochs:
        for batch in dataloader:
            # And all the stuff
            ...
        dataset.shuffle() # after each epoch

Remember about shuffling implementation in such case! (also shuffling can be done inside context manager and you might want to close connection manually or something along those lines).


Instruction: Fast.ai Course 2020: HTTPError: 401 Client Error: PermissionDenied for url

Input:
I am going through the fast.ai course (version 2020) and am currently in lesson 2.
When trying to use the azure key, I always get this error and I can't figure out what to do:
#To download images with Bing Image Search, sign up at Microsoft Azure for a free account. You will be given a key, which you can copy and enter in a cell as follows (replacing 'XXX' with your key and executing it):
key = os.environ.get('AZURE_SEARCH_KEY', 'XXX')

#Once you've set key, you can use search_images_bing. This function is provided by the small utils class included with the notebooks online. If you're not sure where a function is defined, you can just type it in your notebook to find out:
search_images_bing

results = search_images_bing(key, 'grizzly bear')
ims = results.attrgot('content_url')
len(ims)

HTTPError                                 Traceback (most recent call last)
<ipython-input-18-cddb73f3292e> in <module>()
----> 1 results = search_images_bing(key, 'grizzly bear')
      2 ims = results.attrgot('content_url')
      3 len(ims)
1 frames
/usr/local/lib/python3.6/dist-packages/requests/models.py in raise_for_status(self)
    939
    940         if http_error_msg:
--> 941             raise HTTPError(http_error_msg, response=self)
    942
    943     def close(self):
HTTPError: 401 Client Error: PermissionDenied for url:
https://api.bing.microsoft.com/v7.0/images/search?
q=grizzly+bear&count=150&min_height=128&min_width=128

I am using Google Colab.
Maybe I have to add that I don't know what the key name is that I should replace the 'XXX' with. Jeremy Howard said that you would get one when signing up to Azure, but I didn't.

Output: This at first didn't work for me also, then I realized I needed to register for a different Bing Search service on Azure. Look for "Bing Search v7" in the marketplace.