Instruction: Default input dimensions for an image in Neural networks?

Input:
Reading a PyTorch book, I came across this code where the authors change the order of the axis.
img_t.permute(1, 2, 0) (Changes the order of the axes from C × H × W to H × W × C)
Is H x W x C the default input dimensions for an input image in Neural networks?

Output: In PyTorch inputs are bathes in shape N x C x H x W. So N is a batch size, C  is a number of image channels, H and W are height and width as you know. But when you work with for instance cv2, default shape of images is HxWxC so you need to swap dimensions for pytorch.


Instruction: Use of dim=0/1 in pytorch and nn.softmax?

Input:
When using nn.softmax(), we use dim=1 or 0. Here dim=0 should mean row according to intuition but seems it means along the column. Is this true?
>>> x = torch.tensor([[1,2],[3,4]],dtype=torch.float)
>>> F.softmax(x,dim=0)
tensor([[0.1192, 0.1192],
        [0.8808, 0.8808]])
>>> F.softmax(x,dim=1)
tensor([[0.2689, 0.7311],
        [0.2689, 0.7311]])

Here when dim=0, probabilities along the columns sum to 1. Similarly when dim=1 probabilities along the rows sum to 1. Can someone explain how dim is used in PyTorch?

Output:
Indeed, in the 2D case: row refers to axis=0, while column refers to axis=1.
The dim option specifies along which dimension the softmax is apply, i.e. summing back on that same axis will lead to 1s:
>>> x = torch.arange(1, 7, dtype=float).reshape(2,3)
tensor([[1., 2., 3.],
        [4., 5., 6.]], dtype=torch.float64)

On axis=0:
>>> F.softmax(x, dim=0).sum(0)
tensor([1.0000, 1.0000, 1.0000], dtype=torch.float64)

On axis=1:
>>> F.softmax(x, dim=1).sum(1)
>>> tensor([1.0000, 1.0000], dtype=torch.float64)

This is the expected behavior for torch.nn.functional.softmax

[...]
Parameters:

dim (int) – A dimension along which Softmax will be computed (so every slice along dim will sum to 1).


Instruction: Building a Neural Network for Binary Classification on Top of Pre-Trained Embeddings Not Working?

Input:
I am trying to build a Neural Network on top of the embeddings that a pre-trained model outputs. In specific: I have the logits of a base model saved to disk where each example is an array of shape 512 (which originally corresponds to an image) with an associated label (0 or 1). This is what I am doing right now:
Here's the model definition and training loop that I have. Right now it is a simple Linear layer, just to make sure that it works, however, when I run this script, the loss starts at .4 and not ~.7 which is the standard for binary classification. Can anyone spot where I am going wrong?
from transformers.modeling_outputs import SequenceClassifierOutput
class ClassNet(nn.Module):
    def __init__(self, num_labels=2):
        super(ClassNet, self).__init__()
        self.num_labels = num_labels
        self.classifier = nn.Linear(512, num_labels) if num_labels > 0 else nn.Identity()
    def forward(self, inputs):

    logits = self.classifier(inputs)
    loss_fct = nn.CrossEntropyLoss()
    loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
    return SequenceClassifierOutput(
            loss=loss,
            logits=logits
        )

model = ClassNet()
optimizer = optim.Adam(model.parameters(), lr=1e-4,weight_decay=5e-3) #  L2 regularization
loss_fct=nn.CrossEntropyLoss()

for epoch in range(2):  # loop over the dataset multiple times

running_loss = 0.0
for i, data in enumerate(train_loader, 0):
    # get the inputs; data is a list of [inputs, labels]
    #data['embeddings'] -> torch.Size([1, 512])
    #data['labels'] -> torch.Size([1])
    inputs, labels = data['embeddings'], data['labels']

    # zero the parameter gradients
    optimizer.zero_grad()

    # forward + backward + optimize
    outputs = model(inputs)
    loss = loss_fct(outputs.logits.squeeze(1), labels.squeeze())
    loss.backward()
    optimizer.step()

    # print statistics
    running_loss += loss.item()
    if i % 2000 == 1:    # print every 2000 mini-batches
        print('[%d, %5d] loss: %.3f' %
              (epoch + 1, i + 1, running_loss / 2000))
        running_loss = 0.0

an example of printing the outputs.logits.squeeze(1) and labels.squeeze():
#outputs.logits.squeeze(1)
tensor([[-0.2214,  0.2187],
        [ 0.3838, -0.3608],
        [ 0.9043, -0.9065],
        [-0.3324,  0.4836],
        [ 0.6775, -0.5908],
        [-0.8017,  0.9044],
        [ 0.6669, -0.6488],
        [ 0.4253, -0.5357],
        [-1.1670,  1.1966],
        [-0.0630, -0.1150],
        [ 0.6025, -0.4755],
        [ 1.8047, -1.7424],
        [-1.5618,  1.5331],
        [ 0.0802, -0.3321],
        [-0.2813,  0.1259],
        [ 1.3357, -1.2737]], grad_fn=<SqueezeBackward1>)
#labels.squeeze()
tensor([1, 0, 1, 0, 0, 1, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0])
#loss
tensor(0.4512, grad_fn=<NllLossBackward>)

Output:
You are only printing from the second iteration. The above will effectively print for every 200k+1 steps, but i starts at 0
if i % 2000 == 1:    # print every 2000 mini-batches
    print('[%d, %5d] loss: %.3f' %
          (epoch + 1, i + 1, running_loss / 2000))

i.e. one gradient descent step has already occurred. This might be enough to go from the initial loss value -log(1/2) = ~0.69 to the one you observed ~0.45.


Instruction: Difference In the output image when using traced model(.pt) with C++ and OpenCV

Input:
I have retrained the model based on EnlightenGAN. Further I have traced the model in order to execute it in a C++ application using libTorch v1.6. However, I am getting slightly different results as compared to the python(executing the traced model) version.
The model  requires the input RGB tensor and the attention map Image tensor as input. The attention map is basically to inform the model about the image region which requires contrast enhancement.
Below is the code to get inference the output from PT model in python.

def getTransform():
    transform_list = []
    transform_list += [transforms.ToTensor(),
                       transforms.Normalize((0.5, 0.5, 0.5),
                                            (0.5, 0.5, 0.5))]
    return transforms.Compose(transform_list)

def convertToCV(tensor):

    tensor = torch.squeeze(tensor)
    tensor = tensor.cpu().float().detach()
    tensor = torch.unsqueeze(tensor, 0)
    tensor = tensor.permute(1, 2, 0)
    tensor = ((tensor +1)/2.0) * 255.0
    tensor = tensor.numpy()
    return tensor

def proprocess(image):

    transform = getTransform()
    trgbImage = transform(image)
    r,g,b = trgbImage[0]+1, trgbImage[1]+1, trgbImage[2]+1
    tattentionImage = 1. - ((0.299*r+0.587*g+0.114*b)/2.)
    tattentionImage = torch.unsqueeze(tattentionImage, 0)
    trgbImage = torch.unsqueeze(trgbImage, 0)
    tattentionImage = torch.unsqueeze(tattentionImage, 0)

    return trgbImage, tattentionImage

def run(inputPath, OutputPath):

    modelToLoad = torch.jit.load("./EGAN.pt")
    print("OK")
    count =0
    for filename in os.listdir(inputPath):

        print("Processing Image : ", filename)
        inputImage = cv2.imread(os.path.join(inputPath,filename))

        rgbImage, attentionImage = proprocess(inputImage)

        fake, real = modelToLoad.forward(rgbImage,attentionImage )

        fake_B = convertToCV(fake)
        fake_B1 = cv2.cvtColor(fake_B, cv2.COLOR_RGB2BGR)
        cv2.imwrite(OutputPath + "pic1.png" , fake_B )

The C++ version for the inference code is below
#define A 0.299
#define B 0.5870
#define C 0.114

cv::Mat torchTensortoCVMat1C(torch::Tensor& tensor)
{
    tensor = tensor.squeeze(0);
    tensor = tensor.to(torch::kCPU).to(torch::kFloat32).detach();
    tensor = tensor.permute({ 1, 2, 0 }).contiguous();
    tensor = tensor.mul(0.5).add(0.5).mul(255.0);
    tensor = tensor.to(torch::kU8);

    int64_t height = tensor.size(0);
    int64_t width  = tensor.size(1);
    cv::Mat mat    = cv::Mat(cv::Size(width, height), CV_8UC3, tensor.data_ptr<uchar>());
    return mat.clone();
}

std::vector<torch::jit::IValue> CV2Tensor(const cv::Mat& cv_Image)
{
    torch::Tensor tInputImage = (torch::from_blob(cv_Image.data, { cv_Image.rows, cv_Image.cols, cv_Image.channels() }, torch::kByte));
    tInputImage = tInputImage.to(torch::kFloat).div(255);
    tInputImage = tInputImage.sub(0.5).div(0.5).permute({ 2, 0, 1 });

    torch::Tensor red   = tInputImage[0] + 1 ;
    torch::Tensor green = tInputImage[1] + 1 ;
    torch::Tensor blue  = tInputImage[2] + 1;

    red   = red.mul(A);
    green = green.mul(B);
    blue  = blue.mul(C);

    torch::Tensor channelSum = red.add(green).add(blue);
    channelSum = channelSum.div(2.);
    torch::Tensor tGrayImage = 1. - channelSum;

    tGrayImage.unsqueeze_(0);
    tGrayImage.unsqueeze_(0);
    tInputImage.unsqueeze_(0);

    std::vector<torch::jit::IValue> input;
    input.push_back(tInputImage);
    input.push_back(tGrayImage);

    return input;
}

void enhanceImage(const std::string& Img, torch::jit::script::Module& network,const std::string& outputPath, std::string& fileName)
{
    cv::Mat rgbImage;
    cv::Mat inputImage = cv::imread(Img);
    std::vector<torch::jit::IValue> input = CV2Tensor(inputImage);

    try
    {
        auto outputs = network.forward(input).toTuple();
        torch::Tensor resultFake = outputs->elements()[0].toTensor();

        cv::Mat output1 = torchTensortoCVMat(resultFake);
        cv::imshow("out1.png", output1);
        cv::waitKey(0);
    }
    catch (std::exception& e)
    {
        std::cout << e.what() << std::endl;
    }
}

I have also checked the tensor output at all the steps, and they are same. However, after the conversion the output image has color flowing out from brighter regions of input image as show below.
               Python Version
               C++ Version
I have tried many attempt but I am totally puzzeled as to how I should solve the problem. Any help is most welcome.
Thanks.
PS : Let me know if more info is required.

Output:
I found the problem in the above implementation. In the CPP version, I was not clamping the values after doing de-normalisation. I have put a clamping function and now it is working as expected.
The edit part if anyone stumbles on the same problem is below:
tensor = tensor.mul(0.5).add(0.5).mul(255.0); -- > tensor = tensor.mul(0.5).add(0.5).mul(255.0).clamp(0, 255);

Without clamping it as causing overflow in brighter regions of the image.


Instruction: NumPy + PyTorch Tensor assignment

Input:
lets assume we have a tensor representing an image of the shape (910, 270, 1) which assigned a number (some index) to each pixel with width=910 and height=270.
We also have a numpy array of size (N, 3) which maps a 3-tuple to an index.
I now want to create a new numpy array of shape (920, 270, 3) which has a 3-tuple based on the original tensor index and the mapping-3-tuple-numpy array. How do I do this assignment without for loops and other consuming iterations?
This would look simething like:
color_image = np.zeros((self._w, self._h, 3), dtype=np.int32)
self._colors = np.array(N,3) # this is already present
indexed_image = torch.tensor(920,270,1) # this is already present

#how do I assign it to this numpy array?
color_image[indexed_image.w, indexed_image.h] = self._colors[indexed_image.flatten()]

Output:
Assuming you have _colors, and indexed_image. Something that ressembles to:
>>> indexed_image = torch.randint(0, 10, (920, 270, 1))
>>> _colors  = np.random.randint(0, 255, (N, 3))

A common way of converting a dense map to a RGB map is to loop over the label set:
>>> _colors = torch.FloatTensor(_colors)

>>> rgb = torch.zeros(indexed_image.shape[:-1] + (3,))
>>> for lbl in range(N):
...     rgb[lbl == indexed_image[...,0]] = _colors[lbl]


Instruction: How does one install torchtext with cuda >=11.0 (and pytorch 1.9)?

Input:
I've tried multiple things e.g.
conda install -y pytorch==1.9 torchvision torchaudio torchtext cudatoolkit=11.0 -c pytorch -c nvidia

but it never ends up downloading the version with cuda 11.0 or above for some reason.
The error message is too large to paste but you can see details here: https://github.com/pytorch/text/issues/1395
It should be easy to reproduce with an empty env as follow:
conda create -n env_a40 python=3.9
conda activate env_a40
conda install -y pytorch==1.9 torchvision torchaudio torchtext cudatoolkit=11.0 -c pytorch -c nvidia

crossposted:

https://discuss.pytorch.org/t/how-does-one-install-a-torchtext-version-compatible-with-cuda-11-0/132276
https://github.com/pytorch/text/issues/1395

related:

How does one install pytorch 1.9 in an HPC that seems to refuse to cooperate?
https://github.com/pytorch/text/issues/1397

note you can also try it with pip:
pip3 install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html

with no success yet.

Output:
For me it worked with torchtext 0.10.1. The order of how I did things is install pytorch first with:
pip3 install torch==1.9.1+cu111 torchvision==0.10.1+cu111 torchaudio==0.9.1 -f https://download.pytorch.org/whl/torch_stable.html

(probably use the most recent command from the pytorch website https://pytorch.org/get-started/locally/, but if that doesn't work then go to the torchtext website to see what versions of python and pytorch they support and install that. Hopefully in the future torchtext will be in line with the main pytorch branch https://github.com/pytorch/text)
Then since I was using my personal library I installe it in editable mode:
pip install -e ~/ultimate-utils/ultimate-utils-proj-src

or from pypi
pip install ultimate-utils

then go to python to test the pytorch version:
(uutils_env) miranda9~/type-parametric-synthesis $ python
Python 3.9.7 (default, Sep 16 2021, 13:09:58)
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torchtext

>>>
>>> from torchtext.vocab import Vocab
>>>

it knows to install the right version due to my setup.py file. But you can install the right version as follows:
pip install torchtext==0.10.1

In the future the above versions might change and you might have to open an issue in torchtext's github.
Note:
If you are using pip do:
pip3 install torch==1.9.1+cu111 torchvision==0.10.1+cu111 torchaudio==0.9.1 -f https://download.pytorch.org/whl/torch_stable.html
pip3 install torchtext==0.10.1

probably can be compressed to:
pip3 install torch==1.9.1+cu111 torchvision==0.10.1+cu111 torchaudio==0.9.1 torchtext==0.10.1 -f https://download.pytorch.org/whl/torch_stable.html

but have not tried it.
Acks: In particular thanks @ivan for the help!


Instruction: How does one install pytorch 1.9 in an HPC that seems to refuse to cooperate?

Input:
I've been trying to install PyTorch 1.9 with Cuda (ideally 11) on my HPC but I cannot.
The cluster says:
Package typing-extensions conflicts for:
typing-extensions
torchvision -> pytorch==1.8.1 -> typing-extensionsThe following specifications were found to be incompatible with your system:

  - feature:/linux-64::__glibc==2.17=0
  - feature:|@/linux-64::__glibc==2.17=0
  - cffi -> libgcc-ng[version='>=7.5.0'] -> __glibc[version='>=2.17']
  - cudatoolkit=11.0 -> __glibc[version='>=2.17,<3.0.a0']
  - cudatoolkit=11.0 -> libgcc-ng[version='>=7.3.0'] -> __glibc[version='>=2.17']
  - freetype -> libgcc-ng[version='>=7.3.0'] -> __glibc[version='>=2.17']
  - jpeg -> libgcc-ng[version='>=7.5.0'] -> __glibc[version='>=2.17']
  - lcms2 -> libgcc-ng[version='>=7.3.0'] -> __glibc[version='>=2.17']
  - libffi -> libgcc-ng[version='>=7.3.0'] -> __glibc[version='>=2.17']
  - libgcc-ng -> __glibc[version='>=2.17']
  - libmklml -> libgcc-ng -> __glibc[version='>=2.17']
  - libpng -> libgcc-ng[version='>=7.3.0'] -> __glibc[version='>=2.17']
  - libstdcxx-ng -> __glibc[version='>=2.17']
  - libtiff -> libgcc-ng[version='>=7.3.0'] -> __glibc[version='>=2.17']
  - libwebp-base -> libgcc-ng[version='>=7.3.0'] -> __glibc[version='>=2.17']
  - lz4-c -> libgcc-ng[version='>=7.5.0'] -> __glibc[version='>=2.17']
  - mkl-service -> libgcc-ng[version='>=7.5.0'] -> __glibc[version='>=2.17']
  - mkl_fft -> libgcc-ng[version='>=7.3.0'] -> __glibc[version='>=2.17']
  - mkl_random -> libgcc-ng[version='>=7.5.0'] -> __glibc[version='>=2.17']
  - ncurses -> libgcc-ng[version='>=7.3.0'] -> __glibc[version='>=2.17']
  - ninja -> libgcc-ng[version='>=7.3.0'] -> __glibc[version='>=2.17']
  - numpy -> libgcc-ng[version='>=7.5.0'] -> __glibc[version='>=2.17']
  - numpy-base -> libgcc-ng[version='>=7.5.0'] -> __glibc[version='>=2.17']
  - openjpeg -> libgcc-ng[version='>=7.5.0'] -> __glibc[version='>=2.17']
  - openssl -> libgcc-ng[version='>=7.5.0'] -> __glibc[version='>=2.17']
  - pillow -> libgcc-ng[version='>=7.5.0'] -> __glibc[version='>=2.17']
  - python=3.9 -> libgcc-ng[version='>=7.5.0'] -> __glibc[version='>=2.17']
  - pytorch==1.9 -> cudatoolkit[version='>=11.1,<11.2'] -> __glibc[version='>=2.17,<3.0.a0']
  - readline -> libgcc-ng[version='>=7.3.0'] -> __glibc[version='>=2.17']
  - sqlite -> libgcc-ng[version='>=7.5.0'] -> __glibc[version='>=2.17']
  - tk -> libgcc-ng[version='>=7.3.0'] -> __glibc[version='>=2.17']
  - torchvision -> cudatoolkit[version='>=11.1,<11.2'] -> __glibc[version='>=2.17|>=2.17,<3.0.a0']
  - xz -> libgcc-ng[version='>=7.3.0'] -> __glibc[version='>=2.17']
  - zlib -> libgcc-ng[version='>=7.3.0'] -> __glibc[version='>=2.17']
  - zstd -> libgcc-ng[version='>=7.3.0'] -> __glibc[version='>=2.17']

Your installed version is: 2.17

but I don't understand how to use that info to install it. Is it something I can do for the system admins?

When I try to install it with conda, I get a message telling me that it's already installed. However, a conda list greps shows the version is only CPU, not GPU:
(metalearning_gpu) miranda9~/automl-meta-learning $ conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c nvidia

Collecting package metadata (current_repodata.json): done
Solving environment: done

# All requested packages already installed.

(metalearning_gpu) miranda9~/automl-meta-learning $
(metalearning_gpu) miranda9~/automl-meta-learning $ conda list | grep torch
cpuonly                   1.0                           0    pytorch
ffmpeg                    4.3                  hf484d3e_0    pytorch
pytorch                   1.9.0               py3.9_cpu_0  [cpuonly]  pytorch
torch                     1.9.0+cpu                pypi_0    pypi
torchaudio                0.9.0                    pypi_0    pypi
torchmeta                 1.7.0                    pypi_0    pypi
torchvision               0.10.0+cpu               pypi_0    pypi

Attempting to install it with pip completely fails:
(metalearning_gpu) miranda9~/automl-meta-learning $ pip3 install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html

Looking in links: https://download.pytorch.org/whl/torch_stable.html
Collecting torch==1.9.0+cu111
ERROR: Exception:
Traceback (most recent call last):
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 173, in _main
    status = self.run(options, args)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 203, in wrapper
    return func(self, options, args)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 315, in run
    requirement_set = resolver.resolve(
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 94, in resolve
    result = self._result = resolver.resolve(
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 472, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 341, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 172, in _add_to_criteria
    if not criterion.candidates:
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_vendor/resolvelib/structs.py", line 151, in __bool__
    return bool(self._sequence)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 140, in __bool__
    return any(self)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 128, in <genexpr>
    return (c for c in iterator if id(c) not in self._incompatible_ids)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 32, in _iter_built
    candidate = func()
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 204, in _make_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 295, in __init__
    super().__init__(
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 156, in __init__
    self.dist = self._prepare()
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 227, in _prepare
    dist = self._prepare_distribution()
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 305, in _prepare_distribution
    return self._factory.preparer.prepare_linked_requirement(
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 508, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 550, in _prepare_linked_requirement
    local_file = unpack_url(
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 239, in unpack_url
    file = get_http_url(
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 102, in get_http_url
    from_path, content_type = download(link, temp_dir.path)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_internal/network/download.py", line 132, in __call__
    resp = _http_get_download(self._session, link)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_internal/network/download.py", line 115, in _http_get_download
    resp = session.get(target_url, headers=HEADERS, stream=True)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_vendor/requests/sessions.py", line 555, in get
    return self.request('GET', url, **kwargs)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_internal/network/session.py", line 454, in request
    return super().request(method, url, *args, **kwargs)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_vendor/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_vendor/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_vendor/cachecontrol/adapter.py", line 44, in send
    cached_response = self.controller.cached_request(request)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_vendor/cachecontrol/controller.py", line 139, in cached_request
    cache_data = self.cache.get(cache_url)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_internal/network/cache.py", line 54, in get
    return f.read()
MemoryError

Current install script:
## Installation script
# to install do: bash ~/automl-meta-learning/install.sh

#conda update conda

#conda create -y -n metalearning_gpu python=3.9
#conda activate metalearning_gpu
#conda remove --name metalearning_gpu --all

module load cuda-toolkit/11.1
module load gcc/9.2.0

# A40, needs cuda at least 11.0, but 1.9 requires 11
conda activate metalearning_gpu
conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c nvidia
pip3 install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html

#conda activate metalearning_cpu
#conda install pytorch torchvision torchaudio cpuonly -c pytorch
#pip3 install torch==1.9.0+cpu torchvision==0.10.0+cpu torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html

# uutils installs
conda install -y dill
conda install -y networkx>=2.5
conda install -y scipy
conda install -y scikit-learn
conda install -y lark-parser -c conda-forge

# due to compatibility with torch=1.7.x, https://stackoverflow.com/questions/65575871/torchtext-importerror-in-colab
#conda install -y torchtext==0.8.0 -c pytorch

conda install -y tensorboard
conda install -y pandas
conda install -y progressbar2
conda install -y transformers
conda install -y requests
conda install -y aiohttp
conda install -y numpy
conda install -y plotly
conda install -y matplotlib

pip install wandb

# for automl
conda install -y pyyml
conda install -y torchviz
#conda install -y graphviz

#pip install tensorflow
#pip install learn2learn

#pip install -U git+https://github.com/brando90/pytorch-meta.git
#pip install --no-deps torchmeta==1.6.1
pip install --no-deps torchmeta==1.7.0
#        'torch>=1.4.0,<1.9.0',
#        'torchvision>=0.5.0,<0.10.0',
#pip install -y numpy
pip install Pillow
pip install h5py
#pip install requests
pip install ordered-set

pip install higher
#    'torch'

#pip install -U git+https://github.com/moskomule/anatome
pip install --no-deps -U git+https://github.com/moskomule/anatome
#    'torch>=1.9.0',
#    'torchvision>=0.10.0',
pip install tqdm

# - using conda develop rather than pip because uutils installs incompatible versions with the vision cluster
## python -c "import sys; [print(p) for p in sys.path]"
conda install conda-build
# conda develop ~/ultimate-utils/ultimate-utils-proj-src
# conda develop ~/automl-meta-learning/automl-proj-src
# pip install ultimate-utils

# -- extra notes

# local editable installs
# HAL installs, make sure to clone from wmlce 1.7.0 that has h5py ~= 2.9.0 and torch 1.3.1 and torchvision 0.4.2
# pip install torchmeta==1.3.1

Output:
First of all, as @Francois suggested, try to uninstall the CPU only version of pytorch. Also in your installation script, you should use either conda or pip3.
Then you may want to try the following attempts:

using conda: add conda-forge channel  to your command (conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c nvidia -c conda-forge). And make sure conda is updated.
using pip: insert --no-cache-dir into your command (pip3 --no-cache-dir install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html) to avoid the MemoryError.


Instruction: How to do batched dot product in PyTorch?

Input: I have a input tensor that is of size [B, N, 3] and I have a test tensor of size [N, 3] . I want to apply a dot product of the two tensors such that I get [B, N] basically. Is this actually possible?

Output:
Yes, it's possible:
a = torch.randn(5, 4, 3)
b = torch.randn(4, 3)

c = torch.einsum('ijk,jk->ij', a, b) # torch.Size([5, 4])


Instruction: How to train a model with loss calculated by another model in pytorch?

Input:
There are two models A and B. Model A outputs a bike deployment plan for all stations in a city, and model B takes this plan as input and gives the evaluation of each station.
Now, the model B is pretrained, and i want to use the evaluation given by model B as loss to optimize parameters of model A.
Here is the sample code.
A = modelA()
B = modelB()

optimizer = torch.optim.Adam(A.parameters())

def my_loss(deploy):
  shape = deploy.size()
  state = torch.zeros((shape[0], shape[1], 2 + shape[1]), dtype=torch.long)

  # Notice: this step will copy deploy
  state[:, :, 2:] = torch.reshape(deploy, (shape[0], 1, shape[1]))
  state[:, :, 0] = torch.arange(0, shape[1])

  state = torch.reshape(state, (-1, 2 + shape[1]))
  eval = B(state)
  eval = torch.reshape(eval, (shape[0], shape[1]))

  return torch.mean(eval)

# Train model A
for epoch in range(EPOCHS):
  for batch_idx, (x, useless_y) in enumerate(dataloader):
    optimizer.zero_gard()
    pred = A(x)
    loss = my_loss(pred)
    loss.backward()
    optimizer.step()

But in fact, during training, nothing happens, parameters of model A is not updated.
I also tried
optimizer = torch.optim.Adam([{'params': A.parameters()}, {'params': B.parameters(), 'lr':0}])

and nothing happens too.
Any ideas?

Output:
The computational graph is cut off at state, so the loss does not back propagate to A.
Try;
state = torch.zeros((shape[0], shape[1], 2 + shape[1]), dtype=torch.long)
->
state = torch.zeros((shape[0], shape[1], 2 + shape[1]), requires_grad=True) # add requires_grad=True, dtype=torch.long may throw error

However, I still don't think it will work with your code.
Optional suggestion;

I think state_tensors is not defined.
In-place operation state[:, :, 2:] =   may not good. (In my case, this throw error) To copy tensor, .expand() or .repeat(),
and to expand dim, .unsqueeze() may useful to avoid this.


Instruction: ValueError: Target size (torch.Size([128])) must be the same as input size (torch.Size([112]))

Input:
I have a training function, in which inside there are two vectors:
d_labels_a = torch.zeros(128)
d_labels_b = torch.ones(128)

Then I have these features:
# Compute output
features_a = nets[0](input_a)
features_b = nets[1](input_b)
features_c = nets[2](inputs)

And then a domain classifier (nets[4]) makes predictions:
d_pred_a = torch.squeeze(nets[4](features_a))
d_pred_b = torch.squeeze(nets[4](features_b))
d_pred_a = d_pred_a.float()
d_pred_b = d_pred_b.float()
print(d_pred_a.shape)

The error raises in the loss function: `            pred_a = torch.squeeze(nets3)
pred_b = torch.squeeze(nets3)
pred_c = torch.squeeze(nets3)
        loss = criterion(pred_a, labels_a) + criterion(pred_b, labels_b) + criterion(pred_c, labels) + d_criterion(d_pred_a, d_labels_a) + d_criterion(d_pred_b, d_labels_b)

The problem is that d_pred_a/b is different from d_labels_a/b, but only after a certain point. Indeed, when I print the shape of d_pred_a/b it istorch.Size([128])but then it changes totorch.Size([112])` independently.

It comes from here:
# Compute output
features_a = nets[0](input_a)
features_b = nets[1](input_b)
features_c = nets[2](inputs)

because if I print the shape of features_a is torch.Size([128, 2048]) but it changes into torch.Size([112, 2048])
nets[0] is a VGG, like this:
class VGG16(nn.Module):

  def __init__(self, input_size, batch_norm=False):
    super(VGG16, self).__init__()

    self.in_channels,self.in_width,self.in_height = input_size

    self.block_1 = VGGBlock(self.in_channels,64,batch_norm=batch_norm)
    self.block_2 = VGGBlock(64, 128,batch_norm=batch_norm)
    self.block_3 = VGGBlock(128, 256,batch_norm=batch_norm)
    self.block_4 = VGGBlock(256,512,batch_norm=batch_norm)

  @property
  def input_size(self):
      return self.in_channels,self.in_width,self.in_height

  def forward(self, x):

    x = self.block_1(x)
    x = self.block_2(x)
    x = self.block_3(x)
    x = self.block_4(x)
    # x = self.avgpool(x)
    x = torch.flatten(x,1)

    return x

Output: I solved. The problem was the last batch. I used drop_last=True in the dataloader and It worked.


Instruction: How can I solve the problem that method is not iterable?

Input:
I have this Variational autoencoder and I want to use Adam for its optimizer but it has this error I don't know what is wrong here
class VAE(nn.Module):
    def __init__(self):
        super().__init__()

        #encoder
        self.enc = nn.Sequential(
            nn.Linear(1200, 786),
            nn.ReLU(),
            nn.Flatten()
        )
        self.mean = nn.Linear(1200, 2)
        self.log = nn.Linear(1200, 2)
        #decoder
        self.dec = nn.Sequential(
            nn.Linear(2, 1200),
            nn.ReLU(),
        )

    def param(self, mu, Log):
        eps = torch.randn(2, 1200)
        z = mu + (eps * torch.exp(Log * 0.5))
        return z

    def forward(self, x):
        x = self.enc(x)
        mu , log = self.mean(x), self.log(x)
        z = self.param(mu, log)
        x = self.dec(z)
        return x, mu, log

model = VAE()
optim = torch.optim.Adam(model.param, lr=0.01)
criterion = nn.CrossEntropyLoss()

and here is the error
Traceback (most recent call last):
 File "C:\Users\khashayar\PycharmProjects\pythonProject2\VAE.py", line 40, in <module>
   optim = torch.optim.Adam(model.param, lr=0.01)
 File "C:\Users\khashayar\anaconda3\envs\deeplearning\lib\site-packages\torch\optim\adam.py", line 48, in __init__
   super(Adam, self).__init__(params, defaults)
 File "C:\Users\khashayar\anaconda3\envs\deeplearning\lib\site-packages\torch\optim\optimizer.py", line 47, in __init__
   param_groups = list(params)
TypeError: 'method' object is not iterable

how I can solve this?

Output:
The problem is probably in model.param.
param is a method, and as write in the error : "'method' object is not iterable". The optimizer should receive the model parameters, and not the method "param" of the model class.
Try convert optim = torch.optim.Adam(model.param, lr=0.01)  To optim = torch.optim.Adam(model.parameters(), lr=0.01)


Instruction: pytorch custom loss function nn.CrossEntropyLoss

Input:
After studying autograd, I tried to make loss function myself.
And here are my loss
def myCEE(outputs,targets):
    exp=torch.exp(outputs)
    A=torch.log(torch.sum(exp,dim=1))

    hadamard=F.one_hot(targets, num_classes=10).float()*outputs
    B=torch.sum(hadamard, dim=1)
    return torch.sum(A-B)

and I compared with torch.nn.CrossEntropyLoss
here are results
for i,j in train_dl:
    inputs=i
    targets=j
    break

outputs=model(inputs)

myCEE(outputs,targets) : tensor(147.5397, grad_fn=<SumBackward0>)
loss_func = nn.CrossEntropyLoss(reduction='sum')  : tensor(147.5397, grad_fn=<NllLossBackward>)

values were same.
I thought, because those are different functions so  grad_fn are different and it
won't cause any problems.
But something happened!
After 4 epochs, loss values are turned to nan.
Contrary to myCEE, with nn.CrossEntropyLoss learning went well.
So, I wonder  if there is a problem with my function.
After read some posts about nan problems, I stacked more convolutions to the model.
As a result 39-epoch training did not make an error.
Nevertheless, I'd like to know difference between myCEE and nn.CrossEntropyLoss

Output:
torch.nn.CrossEntropyLoss is different to your implementation because it uses a trick to counter instable computation of the exponential when using numerically big values. Given the logits output {l_1, ... l_j, ..., l_n}, the softmax is defined as:
softmax(l_i) = exp(l_i) / sum_j(exp(l_j))

The trick is to multiple both the numerator and denominator by exp(-β):
softmax(l_i) = exp(l_i)*exp(-β) / [sum_j(exp(l_j))*exp(-β)]
             = exp(l_i-β) / sum_j(exp(l_j-β))

Then the log-softmax comes down to:
logsoftmax(l_i) = l_i - β - log[sum_j(exp(l_j-β))]

In practice β is chosen as the highest logit value i.e. β = max_j(l_j).
You can read more about it on this question: Numerically Stable Softmax.


Instruction: Dataloaders Zip

Input:
dataloaders = zip(labeled_trainloader, [None] * len(labeled_trainloader))

What is this line of code means? I  know to zip 2 dataloaders using zip. But why are they adding the length of the dataloader .

Output:
[None] * len(labeled_trainloader) will make a list of None values the same length as labeled_trainloader. For example, [None] * 3 results in [None, None, None].
The zip expression will result in each object in labeled_trainloader being paired with None.
labeled_trainloader = [1, 2, 3]
dataloaders = zip(labeled_trainloader, [None] * len(labeled_trainloader))

print(list(dataloaders))

Output:
[(1, None), (2, None), (3, None)]


Instruction: How to vectorize this computation of a mask for triplet loss function

Input:
Suppose I have a list of numbers lst of length N, along with two numbers epsilon and tau. I want to find the (N,N,N) mask matrix mask such that mask[i][j][k]=1 if and only if
abs(lst[i] - lst[j]) <= epsilon and abs(lst[i] - lst[k]) >= tau
This is what I tried:
d_mat = torch.cdist(lst.unsqueeze(0), lst.unsqueeze(0))
within_eps = torch.where(dmat  <= eps, 1, 0)
over_tau = torch.where(dmat >= tau, 1, 0)

mask = torch.zeros((N,N,N))
for i in range(N):
    for j in range(N):
        for k in range(N):
            if within_eps[i][j] == 1 and over_tau[i][k] == 1:
                mask[i][j][k] = 1
            else:
                mask[i][j][k] = 0

So basically I did it naively. Can you show me, with steps, how you come up with a vectorization for this?

Output:
You successfully created the 2d dmat of pairwise distances. Now you can use torch.logical_and for creating the mask:
mask = torch.logical_and(dmat[..., None] <= eps, dmat[:, None, :] >= tau)

If you want to be explicit about the distance computation (and less efficient) you can:
mask = torch.logical_and(torch.abs(lst[:, None, None] - lst[None, :, None]) <= eps,
                         torch.abs(lst[:, None, None] - lst[None, None, :]) >= tau)


Instruction: TypeError in torch.argmax() when want to find the tokens with the highest `start` score

Input:
I want to run this code for question answering using hugging face transformers.
import torch
from transformers import BertForQuestionAnswering
from transformers import BertTokenizer

#Model
model = BertForQuestionAnswering.from_pretrained('bert-large-uncased-whole-word-masking-finetuned-squad')

#Tokenizer
tokenizer = BertTokenizer.from_pretrained('bert-large-uncased-whole-word-masking-finetuned-squad')

question = '''Why was the student group called "the Methodists?"'''

paragraph = ''' The movement which would become The United Methodist Church began in the mid-18th century within the Church of England.
            A small group of students, including John Wesley, Charles Wesley and George Whitefield, met on the Oxford University campus.
            They focused on Bible study, methodical study of scripture and living a holy life.
            Other students mocked them, saying they were the "Holy Club" and "the Methodists", being methodical and exceptionally detailed in their Bible study, opinions and disciplined lifestyle.
            Eventually, the so-called Methodists started individual societies or classes for members of the Church of England who wanted to live a more religious life. '''

encoding = tokenizer.encode_plus(text=question,text_pair=paragraph)

inputs = encoding['input_ids']  #Token embeddings
sentence_embedding = encoding['token_type_ids']  #Segment embeddings
tokens = tokenizer.convert_ids_to_tokens(inputs) #input tokens

start_scores, end_scores = model(input_ids=torch.tensor([inputs]), token_type_ids=torch.tensor([sentence_embedding]))

start_index = torch.argmax(start_scores)

but I get this error at the last line:
Exception has occurred: TypeError
argmax(): argument 'input' (position 1) must be Tensor, not str
  File "D:\bert\QuestionAnswering.py", line 33, in <module>
    start_index = torch.argmax(start_scores)

I don't know what's wrong. can anyone help me?

Output:
BertForQuestionAnswering returns a QuestionAnsweringModelOutput object.
Since you set the output of BertForQuestionAnswering to start_scores, end_scores, the return QuestionAnsweringModelOutput object is forced convert to a tuple of strings ('start_logits', 'end_logits') causing the type mismatch error.
The following should work:
outputs = model(input_ids=torch.tensor([inputs]), token_type_ids=torch.tensor([sentence_embedding]))

start_index = torch.argmax(outputs.start_logits)


Instruction: what is the best choice for an activation function in case of small sized neural networks

Input:
I am using pytorch and autograd to build my neural network architecture. It is a small 3 layered network with a sinngle input and output. Suppose I have to predict some output function based on some initial conditions and I am using a custom loss function.
The problem I am facing is:

My loss converges initially but gradients vanish eventually.

I have tried sigmoid activation and tanh. tanh gives slightly better results in terms of loss convergence.

I tried using ReLU but since I don't have much weights in my neural network, the weights become dead and it doesn't give good results.

Is there any other activation function apart from sigmoid and tanh that handles the problem of vanishing gradients well enough for small sized neural networks?
Any suggestions on what else can I try

Output:
In the deep learning world, ReLU is usually prefered over other activation functions, because it overcomes the vanishing gradient problem, allowing models to learn faster and perform better. But it could have downsides.
Dying ReLU problem
The dying ReLU problem refers to the scenario when a large number of ReLU neurons only output values of 0. When most of these neurons return output zero, the gradients fail to flow during backpropagation and the weights do not get updated. Ultimately a large part of the network becomes inactive and it is unable to learn further.
What causes the Dying ReLU problem?

High learning rate: If learning rate is set too high, there is a significant chance that new weights will be in negative value range.
Large negative bias: Large negative bias term can indeed cause the inputs to the ReLU activation to become negative.

How to solve the Dying ReLU problem?

Use of a smaller learning rate: It can be a good idea to decrease the learning rate during the training.

Variations of ReLU: Leaky ReLU is a common effective method to solve a dying ReLU problem, and it does so by adding a slight slope in the negative range. There are other variations like PReLU, ELU, GELU. If you want to dig deeper check out this link.

Modification of initialization procedure: It has been demonstrated that the use of a randomized asymmetric initialization can help prevent the dying ReLU problem. Do check out the arXiv paper for the mathematical details

Sources:
Practical guide for ReLU
ReLU variants
Dying ReLU problem


Instruction: I am trying to import:from torchtext.legacy.data import Field, BucketIterator,Iterator,data, but get error 'No module named 'torchtext.legacy'

Input:
I am trying to execute the following code for a nlp proj
import torchtext
from torchtext.legacy.data import Field, BucketIterator, Iterator
from torchtext.legacy import data

----> 6 from torchtext.legacy.data import Field, BucketIterator, Iterator
      7 from torchtext.legacy import data
      8

ModuleNotFoundError: No module named 'torchtext.legacy'.

I have tried it on both kaggle notebook and jupyter notebook and found the same error in both.
i even tried to install !pip install -qqq deepmatcher==0.1.1 in kaggle to solve the issue but it still gives the same error.
is there any solution to this?

Output:
Before you import torchtext.legacy, you need to !pip install torchtext==0.10.0.
Maybe legacy was removed in version 0.11.0.


Instruction: Defining a loss function such that an external array is used

Input:
In my neural network (RNN), I am defining the loss function such that the output of the neural network is used to find the index (binary) and then the index is used to extract the required element from an array which in turn will be used to calculate MSELoss.
However, the program gives parameter().grad = None error which is mostly because the graph is breaking somewhere. What is the problem with the error function defined.
Framework: Pytorch
The codes are as follow:
Neural Network:
class RNN(nn.Module):
  def __init__(self):
    super(RNN, self).__init__()
    self.hidden_size = 8
    # self.input_size = 2
    self.h2o = nn.Linear(self.hidden_size, 1)
    self.h2h = nn.Linear(self.hidden_size, self.hidden_size)
    self.sigmoid = nn.Sigmoid()
  def forward(self,hidden):
    output = self.h2o(hidden)
    output = self.sigmoid(output)
    hidden = self.h2h(hidden)
    return output, hidden
  def init_hidden(self):
    return torch.zeros(1, self.hidden_size)

Loss Function, train step and training
rnn = RNN()
criterion = nn.MSELoss()

def loss_function(previous, output, index):
  code = 2*(output > 0.5).long()
  current = Q_m2[code:code+2, i]
  return criterion(current, previous), current

def train_step():
  hidden = rnn.init_hidden()
  rnn.zero_grad()
  # Q_m2.requires_grad = True
  # Q_m2.create_graph = True
  loss = 0
  previous = Q_m[0:2, 0]
  for i in range(1, samples):
    output, hidden = rnn(hidden)
    l, previous = loss_function(previous, output, i)
    loss+=l
  loss.backward()
  # Q_m2.retain_grad()
  for p in rnn.parameters():
    p.data.add_(p.grad.data, alpha=-0.05)
  return output, loss.item()/(samples - 1)

def training(epochs):
  running_loss = 0
  for i in range(epochs):
    output, loss = train_step()
    print(f'Epoch Number: {i+1}, Loss: {loss}')
    running_loss +=loss

Q_m2
Q_m = np.zeros((4, samples))
for i in range(samples):
  Q_m[:,i] = q_x(U_m[:,i])
Q_m = torch.FloatTensor(Q_m)
Q_m2 = Q_m
Q_m2.requires_grad = True
Q_m2.create_graph = True

Error:
<ipython-input-36-feefd257c97a> in train_step()
     21   # Q_m2.retain_grad()
     22   for p in rnn.parameters():
---> 23     p.data.add_(p.grad.data, alpha=-0.05)
     24   return output, loss.item()/(samples - 1)
     25

AttributeError: 'NoneType' object has no attribute 'data'

Output:
This is a possible solution suggested to me by K. Frank at discuss.pytorch.org

As I read it, code is calculated to be either 0 or 2. You could
instead interpret output (processed appropriately, as necessary) to be
the probability that code should be 0 vs. 2, and then use that
probability to form a weighted average of the 0 and 2 entries in your
Q_m2 array.


Instruction: is there any way to optimize pytorch inference in cpu?

Input:
I am going to serve pytorch model(resnet18) in website.
However, inference in cpu(amd3600) requires 70% of cpu resources.
I don't think the server(heroku) can handle this computation.
Is there any way to optimize inference in cpu?
many thanks

Output:
Admittedly, I'm not an expert on Heroku but probably you can use OpenVINO. OpenVINO is optimized for Intel hardware but it should work with any CPU. It optimizes the inference performance by e.g. graph pruning or fusing some operations together. Here are the performance benchmarks for Resnet-18 converted from PyTorch.
You can find a full tutorial on how to convert the PyTorch model here. Some snippets below.
Install OpenVINO
The easiest way to do it is using PIP. Alternatively, you can use this tool to find the best way in your case.
pip install openvino-dev[pytorch,onnx]

Save your model to ONNX
OpenVINO cannot convert PyTorch model directly for now but it can do it with ONNX model. This sample code assumes the model is for computer vision.
dummy_input = torch.randn(1, 3, IMAGE_HEIGHT, IMAGE_WIDTH)
torch.onnx.export(model, dummy_input, "model.onnx", opset_version=11)

Use Model Optimizer to convert ONNX model
The Model Optimizer is a command line tool which comes from OpenVINO Development Package so be sure you have installed it. It converts the ONNX model to OV format (aka IR), which is a default format for OpenVINO. It also changes the precision to FP16 (to further increase performance). Run in command line:
mo --input_model "model.onnx" --input_shape "[1,3, 224, 224]" --mean_values="[123.675, 116.28 , 103.53]" --scale_values="[58.395, 57.12 , 57.375]" --data_type FP16 --output_dir "model_ir"

Run the inference on the CPU
The converted model can be loaded by the runtime and compiled for a specific device e.g. CPU or GPU (integrated into your CPU like Intel HD Graphics). If you don't know what is the best choice for you, just use AUTO.
# Load the network
ie = Core()
model_ir = ie.read_model(model="model_ir/model.xml")
compiled_model_ir = ie.compile_model(model=model_ir, device_name="CPU")

# Get output layer
output_layer_ir = compiled_model_ir.output(0)

# Run inference on the input image
result = compiled_model_ir([input_image])[output_layer_ir]

Disclaimer: I work on OpenVINO.


Instruction: Why same notebook allocating large different vram in two different environment?

Input:
you can see this notebook is trainable in kaggle using kaggle’s 16gb vram limit : https://www.kaggle.com/firefliesqn/g2net-gpu-newbie
i just tried to run this same notebook locally on rtx3090 gpu where i have torch 1.8 installed and same notebook allocating around 23.3 gb vram,why is this happening and how can i optimize my local environment like kaggle? even if i reduce batch size compared to what is used in kaggle,still locally my notebook allocates around 23gb vram
in kggle i see torch 1.7,tensorflow 2.4 installed and locally as i use rtx3090 so new version of torch and tf is recommended,hence i used torch 1.8.1 and tensorflow 2.6

Output:
By default, TensorFlow allocates max available memory detected.
When using TensorFlow, one can limit the memory used by the following snippet:
gpus = tf.config.experimental.list_physical_devices('GPU')
if gpus:
  # Restrict TensorFlow to only allocate 12GB of memory on the first GPU
  try:
    tf.config.experimental.set_virtual_device_configuration(gpus[0],
        [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=12288)],

where 12228 = 1024x12
Another solution (see discussion below) is to use (works for OP) (use this only if you do not have a specific upper limit of memory to be used) :
 tf.config.experimental.set_memory_growth(physical_devices[0], True)

https://www.tensorflow.org/api_docs/python/tf/config/experimental/set_memory_growth
In PyTorch, this is even easier:
import torch

# use 1/2 memory of the GPU 0 (should allocate very similar amount like TF)
torch.cuda.set_per_process_memory_fraction(0.5, 0)

#Can then check with
total_memory_available = torch.cuda.get_device_properties(0).total_memory


Instruction: Why hasn't a's ID changed?

Input:
a = torch.tensor([1.0, 2.0, 3.0], requires_grad=True)
b = torch.tensor([5.0, 5.0, 5.0])
los = sum(a*b)
los.backward()
print(id(a.grad))
a.grad = torch.tensor([1., 2., 3.])
print(id(a.grad))

The output:
2792503915000
2792503915000

Why didn't the ID of a.grad change?

Output:
Python's id() simply returns a unique number for the object. IDs of objects with non-overlapping lifetimes can be the same; that is, when an object is destroyed, and then a new object is created, the new object can have the same ID as the last. Python object IDs are often compared to memory addresses in C, and (apparently) in some python implementations, they are just memory addresses.
For instance, consider the following snippet:
class a:
    pass

my_a = a()
print(id(my_a))
my_a = a()
print(id(my_a))
my_a = a()
print(id(my_a))

With my particular python implementation, an example output was:
139647922983888
139647922986768
139647922983888

The first ID was repeated in the third instance; this is permissible by the id() api.


Instruction: PyTorch DataLoader returning list instead of tensor on custom Dataset

Input:
I do a trial about my dataset, this is my complete code:
data_root='D:/AuxiliaryDocuments/NYU/'
raw_data_transforms=transforms.Compose([#transforms.ToPILImage(),
                           transforms.CenterCrop((224,101)),
                           transforms.ToTensor()])
depth_data_transforms=transforms.Compose([transforms.CenterCrop((74,55)),
                                      transforms.ToTensor()])

filename_txt={'image_train':'image_train.txt','image_test':'image_test.txt',
          'depth_train':'depth_train.txt','depth_test':'depth_test.txt'}

class Mydataset(Dataset):
  def __init__(self,data_root,transformation,data_type):
     self.transform=transformation
     self.image_path_txt=filename_txt[data_type]
     self.sample_list=list()
     f=open(data_root+'/'+data_type+'/'+self.image_path_txt)
     lines=f.readlines()
     for line in lines:
        line=line.strip()
        line=line.replace(';','')
        self.sample_list.append(line)
     f.close()

  def __getitem__(self, index):
     item=self.sample_list[index]
     img=Image.open(item)
     if self.transform is not None:
        img=self.transform(img)
     idx=index
     print(type(img))
    return idx,img

 def __len__(self):
    return len(self.sample_list)

I print the type of img that is <class 'torch.Tensor'>, then I used the  coding below：
test=Mydataset(data_root,raw_data_transforms,'image_train')
test_1=Mydataset(data_root,depth_data_transforms,'depth_train')
test2=DataLoader(test,batch_size=4,num_workers=0,shuffle=False)
test_2=DataLoader(test_1,batch_size=4,num_workers=0,shuffle=False)

print the information:
for idx,data in enumerate(test_2):
   print(idx,data)
   print(type(data))

but the type of data is '<class 'list'>', which I need is tensor.

Output:
This is the expected output. DataLoader in your case is supposed to return a list. The output of DataLoader is (inputs batch, labels batch).
e.g.
for idx, data in enumerate(test_dataloader):
  if idx == 0:
    print(type(data))
    print(len(data), data[0].shape, data[1].shape)

<class 'list'>
2 torch.Size([64, 1, 28, 28]) torch.Size([64])

Here, the 64 labels corresponds to 64 images in the batch.
In order to pass it to the model, you can do
#If you return img first in your Dataset
    return img, idx

# Either
for idx, data in enumerate(test_dataloader):
    # pass inputs to model
    out = model(data[0])
    # your labels are data[1]

# Or
for idx, (inputs, labels) in enumerate(test_dataloader):
    # pass inputs to model
    out = model(inputs)
    # your labels are in "labels" variable


Instruction: How to test a model before fine-tuning in Pytorch Lightning?

Input:
Doing things on Google Colab.

transformers: 4.10.2
pytorch-lightning: 1.2.7

import torch
from torch.utils.data import DataLoader
from transformers import BertJapaneseTokenizer, BertForSequenceClassification
import pytorch_lightning as pl

dataset_for_loader = [
    {'data':torch.tensor([0,1]), 'labels':torch.tensor(0)},
    {'data':torch.tensor([2,3]), 'labels':torch.tensor(1)},
    {'data':torch.tensor([4,5]), 'labels':torch.tensor(2)},
    {'data':torch.tensor([6,7]), 'labels':torch.tensor(3)},
]
loader = DataLoader(dataset_for_loader, batch_size=2)

for idx, batch in enumerate(loader):
    print(f'# batch {idx}')
    print(batch)

category_list = [
    'dokujo-tsushin',
    'it-life-hack',
    'kaden-channel',
    'livedoor-homme',
    'movie-enter',
    'peachy',
    'smax',
    'sports-watch',
    'topic-news'
]

tokenizer = BertJapaneseTokenizer.from_pretrained(MODEL_NAME)

max_length = 128
dataset_for_loader = []
for label, category in enumerate(tqdm(category_list)):
    # file ./text has lots of articles, categorized by category
    # and they are just plain texts, whose content begins from forth line
    for file in glob.glob(f'./text/{category}/{category}*'):
        lines = open(file).read().splitlines()
        text = '\n'.join(lines[3:])
        encoding = tokenizer(
            text,
            max_length=max_length,
            padding='max_length',
            truncation=True
        )
        encoding['labels'] = label
        encoding = { k: torch.tensor(v) for k, v in encoding.items() }
        dataset_for_loader.append(encoding)

SEED=lambda:0.0

# random.shuffle(dataset_for_loader) # ランダムにシャッフル
random.shuffle(dataset_for_loader,SEED)
n = len(dataset_for_loader)
n_train = int(0.6*n)
n_val = int(0.2*n)
dataset_train = dataset_for_loader[:n_train]
dataset_val = dataset_for_loader[n_train:n_train+n_val]
dataset_test = dataset_for_loader[n_train+n_val:]

dataloader_train = DataLoader(
    dataset_train, batch_size=32, shuffle=True
)
dataloader_val = DataLoader(dataset_val, batch_size=256)
dataloader_test = DataLoader(dataset_test, batch_size=256)

class BertForSequenceClassification_pl(pl.LightningModule):
    def __init__(self, model_name, num_labels, lr):
        super().__init__()
        self.save_hyperparameters()
        self.bert_sc = BertForSequenceClassification.from_pretrained(
            model_name,
            num_labels=num_labels
        )

    def training_step(self, batch, batch_idx):
        output = self.bert_sc(**batch)
        loss = output.loss
        self.log('train_loss', loss)
        return loss

    def validation_step(self, batch, batch_idx):
        output = self.bert_sc(**batch)
        val_loss = output.loss
        self.log('val_loss', val_loss)

    def test_step(self, batch, batch_idx):
        labels = batch.pop('labels')
        output = self.bert_sc(**batch)
        labels_predicted = output.logits.argmax(-1)
        num_correct = ( labels_predicted == labels ).sum().item()
        accuracy = num_correct/labels.size(0)
        self.log('accuracy', accuracy)

    def configure_optimizers(self):
        return torch.optim.Adam(self.parameters(), lr=self.hparams.lr)

checkpoint = pl.callbacks.ModelCheckpoint(
    monitor='val_loss',
    mode='min',
    save_top_k=1,
    save_weights_only=True,
    dirpath='model/',
)
trainer = pl.Trainer(
    gpus=1,
    max_epochs=10,
    callbacks = [checkpoint]
)

model = BertForSequenceClassification_pl(
    MODEL_NAME, num_labels=9, lr=1e-5
)

### (a) ###

# I think this is where I am doing fine-tuning
trainer.fit(model, dataloader_train, dataloader_val)

# this is to score after fine-tuning
test = trainer.test(test_dataloaders=dataloader_test)
print(f'Accuracy: {test[0]["accuracy"]:.2f}')

But I am not really sure how to do a test before fine-tuning, in order to compare two models before and after fine-tuning, in order to show how effective fine-tuning is.
Inserting the following two lines to ### (a) ###:
test = trainer.test(test_dataloaders=dataloader_test)
print(f'Accuracy: {test[0]["accuracy"]:.2f}')

I got this result:
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-13-c8b2c67f2d5c> in <module>()
      9
     10 # 6-19
---> 11 test = trainer.test(test_dataloaders=dataloader_test)
     12 print(f'Accuracy: {test[0]["accuracy"]:.2f}')
     13

/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py in test(self, model, test_dataloaders, ckpt_path, verbose, datamodule)
    896         self.verbose_test = verbose
    897
--> 898         self._set_running_stage(RunningStage.TESTING, model or self.lightning_module)
    899
    900         # If you supply a datamodule you can't supply train_dataloader or val_dataloaders

/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py in _set_running_stage(self, stage, model_ref)
    563         the trainer and the model
    564         """
--> 565         model_ref.running_stage = stage
    566         self._running_stage = stage
    567

AttributeError: 'NoneType' object has no attribute 'running_stage'

I noticed that Trainer.fit() can take None as arguments other than model, so I tried this:
trainer.fit(model)
test=trainer.test(test_dataloaders=dataloader_test)
print(f'Accuracy: {test[0]["accuracy"]:.2f}')

The result:
MisconfigurationException: No `train_dataloader()` method defined. Lightning `Trainer` expects as minimum a `training_step()`, `train_dataloader()` and `configure_optimizers()` to be defined.

Thanks.

Output:
The Trainer needs to call its .fit() in order to set up a lot of things and then only you can do .test() or other methods.
You are right about putting a .fit() just before .test() but the fit call needs to a valid one. You have to feed a dataloader/datamodule to it. But since you don't want to do a training/validation in this fit call, just pass limit_[train/val]_batches=0 while Trainer construction.
trainer = Trainer(gpus=..., ..., limit_train_batches=0, limit_val_batches=0)
trainer.fit(model, dataloader_train, dataloader_val)
trainer.test(model, dataloader_test) # without fine-tuning

The fit call here will just set things up for you and skip training/validation. And then the testing follows. Next time run the same code but without the limit_[train/val]_batches, this will do the pretraining for you
trainer = Trainer(gpus=..., ...)
trainer.fit(model, dataloader_train, dataloader_val)
trainer.test(model, dataloader_test) # with fine-tuning

Clarifying a bit about .fit() taking None for all but model: Its not quite true - you must provide either a DataLoader or a DataModule.


Instruction: BERT text clasisification using pytorch

Input:
I am trying to build a BERT model for text classification with the help of this code [https://towardsdatascience.com/bert-text-classification-using-pytorch-723dfb8b6b5b]. My dataset contains two columns(label, text).
The labels can have three values of (0,1,2). The code works without any error but all values of confusion matrix are 0. Is there something wrong with my code?
import matplotlib.pyplot as plt
import pandas as pd
import torch
from torchtext.data import Field, TabularDataset, BucketIterator, Iterator
import torch.nn as nn
from transformers import BertTokenizer, BertForSequenceClassification
import torch.optim as optim
from sklearn.metrics import accuracy_score, classification_report, confusion_matrix

import seaborn as sns

torch.manual_seed(42)
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

MAX_SEQ_LEN = 128
PAD_INDEX = tokenizer.convert_tokens_to_ids(tokenizer.pad_token)
UNK_INDEX = tokenizer.convert_tokens_to_ids(tokenizer.unk_token)

label_field = Field(sequential=False, use_vocab=False, batch_first=True, dtype=torch.float)
text_field = Field(use_vocab=False, tokenize=tokenizer.encode, lower=False, include_lengths=False, batch_first=True, fix_length=MAX_SEQ_LEN, pad_token=PAD_INDEX, unk_t>
fields = [('label', label_field), ('text', text_field)]
CLASSIFICATION_REPORT = "classification_report.jsonl"

train, valid, test = TabularDataset.splits(path='', train='train.csv', validation='validate.csv', test='test.csv', format='CSV', fields=fields, skip_header=True)

train_iter = BucketIterator(train, batch_size=16, sort_key=lambda x: len(x.text), device=device, train=True, sort=True, sort_within_batch=True)
valid_iter = BucketIterator(valid, batch_size=16, sort_key=lambda x: len(x.text), device=device, train=True, sort=True, sort_within_batch=True)
test_iter = Iterator(test, batch_size=16, device=device, train=False, shuffle=False, sort=False)

class BERT(nn.Module):
        def __init__(self):
                super(BERT, self).__init__()
                options_name = "bert-base-uncased"
                self.encoder = BertForSequenceClassification.from_pretrained(options_name, num_labels = 3)

        def forward(self, text, label):
                loss, text_fea = self.encoder(text, labels=label)[:2]
                return loss, text_fea

def train(model, optimizer, criterion = nn.BCELoss(), train_loader = train_iter, valid_loader = valid_iter, num_epochs = 5, eval_every = len(train_iter) // 2, file_pat>        running_loss = 0.0
        valid_running_loss = 0.0
        global_step = 0
        train_loss_list = []
        valid_loss_list = []
        global_steps_list = []

        model.train()

        for epoch in range(num_epochs):
                for (label, text), _ in train_loader:
                        label = label.type(torch.LongTensor)
                        label = label.to(device)
                        text = text.type(torch.LongTensor)
                        text = text.to(device)
                        output = model(text, label)
                        loss, _ = output
                        optimizer.zero_grad()
                        loss.backward()
                        optimizer.step()
                        running_loss += loss.item()
                        global_step += 1
                        if global_step % eval_every == 0:
                                model.eval()
                                with torch.no_grad():
                                        for (label, text), _ in valid_loader:
                                                label = label.type(torch.LongTensor)
                                                label = label.to(device)
                                                text = text.type(torch.LongTensor)
                                                text = text.to(device)
                                                output = model(text, label)
                                                loss, _ = output
                                                valid_running_loss += loss.item()

                                average_train_loss = running_loss / eval_every
                                average_valid_loss = valid_running_loss / len(valid_loader)
                                train_loss_list.append(average_train_loss)
                                valid_loss_list.append(average_valid_loss)
                                global_steps_list.append(global_step)

                                # resetting running values
                                running_loss = 0.0
                                valid_running_loss = 0.0
                                model.train()

                                # print progress
                                print('Epoch [{}/{}], Step [{}/{}], Train Loss: {:.4f}, Valid Loss: {:.4f}'.format(epoch+1, num_epochs, global_step, num_epochs*len(tra>
                                if best_valid_loss > average_valid_loss:
                                        best_valid_loss = average_valid_loss
        print('Finished Training!')

model = BERT().to(device)
optimizer = optim.Adam(model.parameters(), lr=2e-5)

train(model=model, optimizer=optimizer)

def evaluate(model, test_loader):
        y_pred = []
        y_true = []
        model.eval()
        with torch.no_grad():
                for (label, text), _ in test_loader:
                        label = label.type(torch.LongTensor)
                        label = label.to(device)
                        text = text.type(torch.LongTensor)
                        text = text.to(device)
                        output = model(text, label)

                        _, output = output
                        y_pred.extend(torch.argmax(output, 2).tolist())
                        y_true.extend(label.tolist())
        print('Classification Report:')
        print(classification_report(y_true, y_pred, labels=[0,1,2], digits=4))
best_model = BERT().to(device)
evaluate(best_model, test_iter)

Output: you are using criterion = nn.BCELoss(), binary cross entropy for a multi class classification problem, "the labels can have three values of (0,1,2)". use suitable loss function for multiclass classification.


Instruction: Resize feature vector from neural network

Input:
I am trying to perform a task of approximation of two embeddings (textual and visual).
For the visual embedding, I am using VGG as the encoder. The output is a 1x1000 embedding. For the textual encoder, I am using a Transformer to which output is shaped 1x712. What I want is to convert both these vectors to the same dimension 512.
img_features.shape, txt_features.shape = (1,1000),(1,712)

How can I do it in PyTorch?
Add a final layer in each architecture that models the output to 512?

Output:
You could either apply a differentiable PCA operator such as torch.pca_lowrank.

Alternatively, an easier solution is to use two fully connected adapter layers to learn two mappings. One for you image features 1000 -> n, the other for textual features: 712 -> n. Then you can choose a fusion strategy to combine the  two features shaped (1,n): either using concatenation, point-wise addition/multiplication (in thoses cases n should be equal to 512. Esle you can learn a final mapping n*2 -> 512.


Instruction: create a linear model with fixed weights in Pytorch

Input:
I want to create a linear network with a single layer under PyTorch, but I want the weights to be manually initialized and to remain fixed.
For example the values of the weights with the model:
layer = nn.Linear(4, 1, bias=False)
weights = tensor([[ 0.6],
                  [0.25],
                  [ 0.1],
                  [0.05]], dtype=torch.float64)

Is this achievable? If so, how can I do it? Or is there an alternative linear function?

Output:
You can freeze your layer by setting the requires_grad to False:
layer.requires_grad_(False)

This way the gradients of the layer's parameters won't get computed.
Or by directly defining so when initializing the parameter:
layer = nn.Linear(4, 1, bias=False)
layer.weight = nn.Parameter(weights, requires_grad=False)

Alternatively, given an input x shaped (n, 4), you can compute the result with a simple matrix multiplication as:
>>> x@weights
# equivalent to torch.matmul(x, weights)


Instruction: How can I use different encoder and decoder transformers models

Input:
simply input is image ===> output text(feature extractor )
I want to use separate encoder and decoder models for Handwriting recognition TrOCR shows an error that the input image is diff size for each model How can I modify the config of model or do normalize fro input image to models
from transformers import (
    TrOCRConfig,
    TrOCRProcessor,
    TrOCRForCausalLM,
    ViTConfig,
    ViTModel,
    VisionEncoderDecoderModel,
)
import requests
import cv2
from PIL import Image

# TrOCR is a decoder model and should be used within a VisionEncoderDecoderModel
# init vision2text model with random weights
encoder = ViTModel(ViTConfig())
decoder = TrOCRForCausalLM(TrOCRConfig())
model = VisionEncoderDecoderModel(encoder=encoder, decoder=decoder)

# If you want to start from the pretrained model, load the checkpoint with `VisionEncoderDecoderModel`
processor = TrOCRProcessor.from_pretrained("microsoft/trocr-base-handwritten")
# model = VisionEncoderDecoderModel.from_pretrained("microsoft/trocr-base-handwritten")
tokenizer=processor.feature_extractor
# load image from the IAM dataset
url = "https://fki.tic.heia-fr.ch/static/img/a01-122-02.jpg"
image = Image.open(requests.get(url, stream=True).raw).convert("RGB")
# image normlize with 224 x 224
image_0 = cv2.imread('/content/mm.png')
pixel_values = processor(image_0, return_tensors="pt").pixel_values
text = "industry, ' Mr. Brown commented icily. ' Let us have a"

# training
model.config.decoder_start_token_id = processor.tokenizer.cls_token_id
model.config.pad_token_id = processor.tokenizer.pad_token_id
model.config.vocab_size = model.config.decoder.vocab_size
model.config.encoder.image_size = 224
# model.config.image_size  = 384
labels = processor.tokenizer(text, return_tensors="pt").input_ids
outputs = model(pixel_values, labels=labels)
loss = outputs.loss
round(loss.item(), 2)

# inference
generated_ids = model.generate(pixel_values)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
generated_text

And this is the error I got
/usr/local/lib/python3.7/dist-packages/transformers/models/vit/modeling_vit.py in forward(self, pixel_values, interpolate_pos_encoding)
171             if height != self.image_size[0] or width != self.image_size[1]:
172                 raise ValueError(
--> 173                     f"Input image size ({height}{width}) doesn't match model"
174                     f" ({self.image_size[0]}{self.image_size[1]})."
175                 )
ValueError: Input image size (384384) doesn't match model (224224).

Output:
I think the way you've worded your question doesn't line up with the example you've given. Firstly, the example array you've given is 3D, not 2D. You can do
>>> arr.shape
(1,2,3)
>>> arr.ndim
3

Presumably this is a mistake, and you want your array to be 2D, so you would do
arr = np.array([[5., 2., -5.], [4., 3., 1.]])

instead.
Secondly, if a and b are values that, if an element is between then to set that element to value c rather than a and b being indexes, then the np.where function is great for this.
def overwrite_interval(arr , a , b , c):
    inds = np.where((arr >= a) * (arr <= b))
    arr[inds] = c
    return arr

np.where returns a tuple, so sometimes it can be easier to work with boolean arrays directly. In which case, the function would look like this
def overwrite_interval(arr , a , b , c):
    inds = (arr >= a) * (arr <= b)
    arr[inds] = c
    return arr

Does this work for you, and is this your intended meaning? Note that the solution I've provided would work as is if you still meant for the initial array to be a 3D array.


Instruction: Total Variation Regularization for Tensors in Python

Input:
Formula
Hi, I am trying to implement total variation function for tensor or in more accurate, multichannel images. I found that for above Total Variation (in picture), there is source code like this:
def compute_total_variation_loss(img, weight):
    tv_h = ((img[:,:,1:,:] - img[:,:,:-1,:]).pow(2)).sum()
    tv_w = ((img[:,:,:,1:] - img[:,:,:,:-1]).pow(2)).sum()
    return weight * (tv_h + tv_w)

Since, I am very beginner in python I didn't understood how the indices are referred to i and j in image. I also want to add total variation for c (besides i and j) but I don't know which index refers to c.
Or to be more concise, how to write following equation in python:
enter image description here

Output:
This function assumes batched images. So img is a 4 dimensional tensor of dimensions (B, C, H, W) (B is the number of images in the batch, C the number of color channels, H the height and W the width).
So, img[0, 1, 2, 3] is the pixel (2, 3) of the second color (green in RGB) in the first image.
In Python (and Numpy and PyTorch), a slice of elements can be selected with the notation i:j, meaning that the elements i, i + 1, i + 2, ..., j - 1 are selected. In your example, : means all elements, 1: means all elements but the first and :-1 means all elements but the last (negative indices retrieves the elements backward). Please refer to tutorials on "slicing in NumPy".
So img[:,:,1:,:] - img[:,:,:-1,:] is equivalent to the (batch of) images minus themselves shifted by one pixel vertically, or, in your notation X(i + 1, j, k) - X(i, j, k). Then the tensor is squared (.pow(2)) and summed (.sum()). Note that the sum is also over the batch in this case, so you receive the total variation of the batch, not of each images.


Instruction: Pytorch function calling

Input:
I am trying to compute a trignometric function using pytorch, but having issues while calling it via function, below is my code:-
def func(x,y):
  return torch.exp(torch.sin(x)/x-y)

func(torch.tensor[2,3])

Error:-
TypeError - Traceback (most recent call last)
<ipython-input-16-beb818f912f5> in <module>()
----> 1 func(torch.tensor([2, 3]))

TypeError: newf() missing 1 required positional argument: 'y'

What is incorrect in this code while calling the function?

Output:
You need to do it with unpacking:
func(*torch.tensor[2,3])


Instruction: Where is torch.matmul implemented?

Input:
Where is torch.matmul implemented, especially the part that runs on the GPU?
The whole project is 2M lines of code. I tried to grep the sources of the 1.8.2 release, but have trouble finding this function. I'm guessing it's generated from something...

Output:
It should be implemented below:
https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/LinearAlgebra.cpp#L1450
The matmul function included by the Functions.h and ATen.h included Functions.h
And inside pytorch api(pytorch/torch/csrc/api/include/torch/linalg.h), it includes <ATen/ATen.h>
So:
pytorch/torch/csrc/api/include/torch/linalg.h > ATen > Functions > LinearAlgebra > matmul


Instruction: Why does numpy and pytorch give different results after mean and variance normalization?

Input:
I am working on a problem in which a matrix has to be mean-var normalized row-wise. It is also required that the normalization is applied after splitting each row into tiny batches.
The code seem to work for Numpy, but fails with Pytorch (which is required for training).
It seems Pytorch and Numpy results differ. Any help will be greatly appreciated.
Example code:
import numpy as np
import torch

def normalize(x, bsize, eps=1e-6):
    nc = x.shape[1]
    if nc % bsize != 0:
        raise Exception(f'Number of columns must be a multiple of bsize')
    x = x.reshape(-1, bsize)
    m = x.mean(1).reshape(-1, 1)
    s = x.std(1).reshape(-1, 1)
    n = (x - m) / (eps + s)
    n = n.reshape(-1, nc)
    return n

# numpy
a = np.float32(np.random.randn(8, 8))
n1 = normalize(a, 4)
# torch
b = torch.tensor(a)
n2 = normalize(b, 4)
n2 = n2.numpy()

print(abs(n1-n2).max())

Output:
In the first example you are calling normalize with a, a numpy.ndarray, while in the second you call normalize with b, a torch.Tensor.
According to the documentation page of torch.std, Bessel’s correction is used by default to measure the standard deviation. As such the default behavior between numpy.ndarray.std and torch.Tensor.std is different.

If unbiased is True, Bessel’s correction will be used. Otherwise, the sample deviation is calculated, without any correction.

torch.std(input, dim, unbiased, keepdim=False, *, out=None) → Tensor
Parameters

input (Tensor) – the input tensor.
unbiased (bool) – whether to use Bessel’s correction (δN = 1).

You can try yourself:
>>> a.std(), b.std(unbiased=True), b.std(unbiased=False)
(0.8364538, tensor(0.8942), tensor(0.8365))


Instruction: 'RuntimeError: mat1 and mat2 shapes cannot be multiplied', how do I solve it?

Input:
I'm trying to implement a ResNet1D, that should take as input a window of ECG signal ,containing a single heart beat, in my case with size 950 samples, and I want to predict the length of the QRS interval.
Here's the code for the network implementation:
class Bottleneck(nn.Module):
    expansion = 4

    def __init__(self, in_planes, planes, stride=1):
        super(Bottleneck, self).__init__()

        self.conv1 = nn.Conv1d(in_planes, planes, kernel_size=1, bias=False)
        self.bn1 = nn.BatchNorm1d(planes)

        self.conv2 = nn.Conv1d(planes, planes, kernel_size=3, stride=stride, padding=1, bias=False)
        self.bn2 = nn.BatchNorm1d(planes)

        self.conv3 = nn.Conv1d(planes,    self.expansion*planes, kernel_size=1, bias=False)
        self.bn3 = nn.BatchNorm1d(self.expansion*planes)

        self.shortcut = nn.Sequential()
        if stride != 1 or in_planes != self.expansion*planes:
            self.shortcut = nn.Sequential(
                nn.Conv1d(in_planes, self.expansion*planes, kernel_size=1, stride=stride, bias=False),
                nn.BatchNorm1d(self.expansion*planes)
                )

    def forward(self, x):
        out = F.relu(self.bn1(self.conv1(x)))
        out = F.relu(self.bn2(self.conv2(out)))
        out = self.bn3(self.conv3(out))
        out += self.shortcut(x)
        out = F.relu(out)
        return out

class ResNet(nn.Module):
    def __init__(self, block, num_blocks, num_classes=3):
        super(ResNet, self).__init__()
        self.in_planes = 64

        self.avg1 = nn.AvgPool1d(1024, stride=2)
        self.conv1 = nn.Conv1d(1, 128, kernel_size=3, stride=1, padding=1, bias=False)
        self.bn1 = nn.BatchNorm1d(128)

        self.layer1 = self._make_layer(block, 128, num_blocks[0], stride=1)
        self.layer2 = self._make_layer(block, 256, num_blocks[1], stride=2)
        self.layer3 = self._make_layer(block, 512, num_blocks[2], stride=2)
        self.layer4 = self._make_layer(block, 1024, num_blocks[3], stride=2)
        self.linear1 = nn.Linear(19968*block.expansion, 1024)
        self.linear2 = nn.Linear(1024, num_classes)

    def _make_layer(self, block, planes, num_blocks, stride):
        strides = [stride] + [1]*(num_blocks-1)
        layers = []
        for stride in strides:
            layers.append(block(self.in_planes, planes, stride))
            self.in_planes = planes * block.expansion

        return nn.Sequential(*layers)

    def forward(self, x):
        out = self.avg1(x)
        out = F.rel(self.bn1(self.conv1(out)))

        out = self.layer1(out)
        out = self.layer2(out)
        out = self.layer3(out)
        out = self.layer4(out)

        out = F.avg_pool1d(out, 16)
        out = out.view(out.size(0), -1)
        out = self.linear1(out)
        out = self.linear2(out)

        return out

    def ResNet50():
        return ResNet(Bottleneck, [3, 4, 6, 3], num_classes=1)

The input that I'm feeding to the network is a dataloader with batch size = 32, number of channels = 1 and sample length = 950.
When training the network I get this error:
RuntimeError: mat1 and mat2 shapes cannot be multiplied (1x3584 and 19968x1024)

I get that the error is in the Linear layer, but I don't understand how I should change the dimensions in order to make it work. Can you please explain this to me?

Output:
Matrices are multiplicable if the number of columns of the first matrix matches the number of rows of the second one. For instance, MxN * NxK. So here you have wrong shapes. You can always calculate the sizes of the outputs of each layer by yourself to make sure if your shapes are correct.
So here I think it should work if you change this

self.linear1 = nn.Linear(19968*block.expansion, 1024)

to this

self.linear1 = nn.Linear(3584*block.expansion, 1024)


Instruction: Can this matrix calculation be implemented or approximated without an intermediate 3D matrix?

Input:
Given an NxN matrix W, I'm looking to calculate an NxN matrix C given by the equation in this link: https://i.stack.imgur.com/dY7rY.png, or in LaTeX
$$C_{ij} = \max_k \bigg\{ \sum_l \bigg( W_{ik}W_{kl}W_{lj} - W_{ik}W_{kj} \bigg)  \bigg\}.$$

I have tried to implement this in PyTorch but I've either encountered memory problems by constructing an intermediate NxNxN 3D matrix which, for large N, causes my GPU to run out of memory, or used a for-loop over k which is then very slow. I can't work out how I can get round these. How might I implement this calculation, or an approximation of it, without a large intermediate matrix like this?
Suggestions, pseudocode in any language or an implementation in any of Python/Numpy/PyTorch would be much appreciated.

Output:
The first solution using Numba (You can do the same using Cython or plain C) would be to formulate the problem using simple loops.
import numpy as np
import numba as nb

@nb.njit(fastmath=True,parallel=True)
def calc_1(W):
    C=np.empty_like(W)
    N=W.shape[0]

    for i in nb.prange(N):
        TMP=np.empty(N,dtype=W.dtype)
        for j in range(N):
            for k in range(N):
                acc=0
                for l in range(N):
                    acc+=W[i,k]*W[k,l]*W[l,j]-W[i,k]*W[k,j]
                TMP[k]=acc
            C[i,j]=np.max(TMP)
    return C

Francesco provided a simplification which scales far better for larger array sizes. This leads to the following, where I also optimized away a small temporary array.
@nb.njit(fastmath=True,parallel=True)
def calc_2(W):
    C=np.empty_like(W)
    N=W.shape[0]
    M = np.dot(W,W) - N * W

    for i in nb.prange(N):
        for j in range(N):
            val=W[i,0]*M[0,j]
            for k in range(1,N):
                TMP=W[i,k]*M[k,j]
                if TMP>val:
                    val=TMP
            C[i,j]=val
    return C

This can be optimized further by partial loop unrolling and optimizing the array access. Some compilers may do this automatically.
@nb.njit(fastmath=True,parallel=True)
def calc_3(W):
    C=np.empty_like(W)
    N=W.shape[0]
    W=np.ascontiguousarray(W)
    M = np.dot(W.T,W.T) - W.shape[0] * W.T

    for i in nb.prange(N//4):
        for j in range(N):
            val_1=W[i*4+0,0]*M[j,0]
            val_2=W[i*4+1,0]*M[j,0]
            val_3=W[i*4+2,0]*M[j,0]
            val_4=W[i*4+3,0]*M[j,0]
            for k in range(1,N):
                TMP_1=W[i*4+0,k]*M[j,k]
                TMP_2=W[i*4+1,k]*M[j,k]
                TMP_3=W[i*4+2,k]*M[j,k]
                TMP_4=W[i*4+3,k]*M[j,k]
                if TMP_1>val_1:
                    val_1=TMP_1
                if TMP_2>val_2:
                    val_2=TMP_2
                if TMP_3>val_3:
                    val_3=TMP_3
                if TMP_4>val_4:
                    val_4=TMP_4

            C[i*4+0,j]=val_1
            C[i*4+1,j]=val_2
            C[i*4+2,j]=val_3
            C[i*4+3,j]=val_4

    #Remainder
    for i in range(N//4*4,N):
        for j in range(N):
            val=W[i,0]*M[j,0]
            for k in range(1,N):
                TMP=W[i,k]*M[j,k]
                if TMP>val:
                    val=TMP
            C[i,j]=val
    return C

Timings
W=np.random.rand(100,100)
%timeit calc_1(W)
#16.8 ms ± 131 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)
%timeit calc_2(W)
#449 µs ± 25.7 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)
%timeit calc_3(W)
#259 µs ± 47.4 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)

W=np.random.rand(2000,2000)
#Temporary array would be 64GB in this case
%timeit calc_2(W)
#5.37 s ± 174 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%timeit calc_3(W)
#596 ms ± 30.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)


Instruction: How to convert tokenized words back to the original ones after inference?

Input:
I'm writing a inference script for already trained NER model, but I have trouble with converting encoded tokens (their ids) into original words.
# example input
df = pd.DataFrame({'_id': [1], 'body': ['Amazon and Tesla are currently the best picks out there!']})

# calling method that handles inference:
ner_model = NER()
ner_model.recognize_from_df(df, 'body')

# here is only part of larger NER class that handles the inference:
def recognize_from_df(self, df: pd.DataFrame, input_col: str):
    predictions = []
    df = df[['_id', input_col]].copy()
    dataset = Dataset.from_pandas(df)
    # tokenization, padding, truncation:
    encoded_dataset = dataset.map(lambda examples: self.bert_tokenizer(examples[input_col],
                                      padding='max_length', truncation=True, max_length=512), batched=True)
    encoded_dataset.set_format(type='torch', columns=['input_ids', 'attention_mask'], device=device)
    dataloader = torch.utils.data.DataLoader(encoded_dataset, batch_size=32)
    encoded_dataset_ids = encoded_dataset['_id']

    for batch in dataloader:
        output = self.model(**batch)
        # decoding predictions and tokens
        for i in range(batch['input_ids'].shape[0]):
            tags = [self.unique_labels[label_id] for label_id in output[i]]
            tokens = [t for t in self.bert_tokenizer.convert_ids_to_tokens(batch['input_ids'][i]) if t != '[PAD]']
        ...

The results are close to what I need:
# tokens:
['[CLS]', 'am', '##az', '##on', 'and', 'te', '##sla', 'are', 'currently', 'the', 'best', 'picks', 'out', 'there', ...]
# tags:
['X', 'B-COMPANY', 'X', 'X', 'O', 'B-COMPANY', 'X', 'O', 'O', 'O', 'O', 'O', 'O', 'O', ...]

How to combine 'am', '##az', '##on' and 'B-COMPANY', 'X', 'X' into one token/tag? I know that there is a method called convert_tokens_to_string in Tokenizer, but it returns just one big string, which is hard to map to tag.
Regards

Output:
Provided you only want to "merge" company names one could do that in a linear time with pure Python.
Skipping the beginning of sentence token [CLS] for brevity:
tokens = tokens[1:]
tags = tags[1:]

The function below will merge company tokens and increase pointer appropriately:
def merge_company(tokens, tags):
    generated_tokens = []
    i = 0
    while i < len(tags):
        if tags[i] == "B-COMPANY":
            company_token = [tokens[i]]
            for j in range(i + 1, len(tags)):
                i += 1
                if tags[j] != "X":
                    break
                else:
                    company_token.append(tokens[j][2:])
            generated_tokens.append("".join(company_token))
        else:
            generated_tokens.append(tokens[i])
            i += 1

    return generated_tokens

Usage is pretty simple, please notice tags need their Xs removed as well though:
tokens = merge_company(tokens, tags)
tags = [tag for tag in tags if tag != "X"]

This would give you:
['amazon', 'and', 'tesla', 'are', 'currently', 'the', 'best', 'picks', 'out', 'there']
['B-COMPANY', 'O', 'B-COMPANY', 'O', 'O', 'O', 'O', 'O', 'O', 'O']


Instruction: How to extract loss and accuracy from logger by each epoch in pytorch lightning?

Input:
I want to extract all data to make the plot, not with tensorboard. My understanding is all log with loss and accuracy is stored in a defined directory since tensorboard draw the line graph.
%reload_ext tensorboard
%tensorboard --logdir lightning_logs/

However, I wonder how all log can be extracted from the logger in pytorch lightning. The next is the code example in training part.
#model
ssl_classifier = SSLImageClassifier(lr=lr)

#train
logger = pl.loggers.TensorBoardLogger(name=f'ssl-{lr}-{num_epoch}', save_dir='lightning_logs')

trainer = pl.Trainer(progress_bar_refresh_rate=20,
                            gpus=1,
                            max_epochs = max_epoch,
                            logger = logger,
                            )

trainer.fit(ssl_classifier, train_loader, val_loader)

I had confirmed that trainer.logger.log_dir returned directory which seems to save logs and trainer.logger.log_metrics returned <bound method TensorBoardLogger.log_metrics of <pytorch_lightning.loggers.tensorboard.TensorBoardLogger object at 0x7efcb89a3e50>>.
trainer.logged_metrics returned only the log in the final epoch, like
{'epoch': 19,
 'train_acc': tensor(1.),
 'train_loss': tensor(0.1038),
 'val_acc': 0.6499999761581421,
 'val_loss': 1.2171183824539185}

Do you know how to solve the situation?

Output:
Lightning do not store all logs by itself. All it does is streams them into the logger instance and the logger decides what to do.
The best way to retrieve all logged metrics is by having a custom callback:
class MetricTracker(Callback):

  def __init__(self):
    self.collection = []

  def on_validation_batch_end(trainer, module, outputs, ...):
    vacc = outputs['val_acc'] # you can access them here
    self.collection.append(vacc) # track them

  def on_validation_epoch_end(trainer, module):
    elogs = trainer.logged_metrics # access it here
    self.collection.append(elogs)
    # do whatever is needed

You can then access all logged stuff from the callback instance
cb = MatricTracker()
Trainer(callbacks=[cb])

cb.collection # do you plotting and stuff


Instruction: Understanding the architecture of an LSTM for sequence classification

Input:
I have this model in pytorch that I have been using for sequence classification.
class RoBERT_Model(nn.Module):

    def __init__(self, hidden_size = 100):
        self.hidden_size = hidden_size
        super(RoBERT_Model, self).__init__()
        self.lstm = nn.LSTM(768, hidden_size, num_layers=1, bidirectional=False)
        self.out = nn.Linear(hidden_size, 2)

    def forward(self, grouped_pooled_outs):
        # chunks_emb = pooled_out.split_with_sizes(lengt) # splits the input tensor into a list of tensors where the length of each sublist is determined by length

        seq_lengths = torch.LongTensor([x for x in map(len, grouped_pooled_outs)]) # gets the length of each sublist in chunks_emb and returns it as an array

        batch_emb_pad = nn.utils.rnn.pad_sequence(grouped_pooled_outs, padding_value=-91, batch_first=True) # pads each sublist in chunks_emb to the largest sublist with value -91
        batch_emb = batch_emb_pad.transpose(0, 1)  # (B,L,D) -> (L,B,D)
        lstm_input = nn.utils.rnn.pack_padded_sequence(batch_emb, seq_lengths, batch_first=False, enforce_sorted=False) # seq_lengths.cpu().numpy()

        packed_output, (h_t, h_c) = self.lstm(lstm_input, )  # (h_t, h_c))
        # output, _ = nn.utils.rnn.pad_packed_sequence(packed_output, padding_value=-91)
        h_t = h_t.view(-1, self.hidden_size) # (-1, 100)

        return self.out(h_t) # logits

The issue that I am having is that I am not entirely convinced of what data is being passed to the final classification layer. I believe what is being done is that only the final LSTM cell in the last layer is being used for classification. That is there are hidden_size features that are passed to the feedforward layer.
I have depicted what I believe is going on in this figure here:

Is this understanding correct? Am I missing anything?
Thanks.

Output:
Your code is a basic LSTM for classification, working with a single rnn layer.
In your picture you have multiple LSTM layers, while, in reality, there is only one, H_n^0 in the picture.

Your input to LSTM is of shape (B, L, D) as correctly pointed out in the comment.
packed_output and h_c is not used at all, hence you can change this line to: _, (h_t, _) = self.lstm(lstm_input) in order no to clutter the picture further
h_t is output of last step for each batch element, in general (B, D * L, hidden_size). As this neural network is not bidirectional D=1, as you have a single layer L=1 as well, hence the output is of shape (B, 1, hidden_size).
This output is reshaped into nn.Linear compatible (this line: h_t = h_t.view(-1, self.hidden_size)) and will give you output of shape (B, hidden_size)
This input is fed to a single nn.Linear layer.

In general, the output of the last time step from RNN is used for each element in the batch, in your picture H_n^0 and simply fed to the classifier.
By the way, having self.out = nn.Linear(hidden_size, 2) in classification is probably counter-productive; most likely your are performing binary classification and self.out = nn.Linear(hidden_size, 1) with torch.nn.BCEWithLogitsLoss might be used. Single logit contains information whether the label should be 0 or 1; everything smaller than 0 is more likely to be 0 according to nn, everything above 0 is considered as a 1 label.


Instruction: How to obtain the path name of train_dataset after using random_split in torch

Input:
I have the following code:
import torch, torchvision
root_dataset ="./data"
dataset = torchvision.datasets.folder.ImageFolder(root=root_dataset, transform=None, target_transform=None)
train_dataset, valid_dataset = torch.utils.data.dataset.random_split(
    dataset=dataset,
    lengths=[num_train, num_valid]
)

My question is:
How can I obtain the name list of the path of train_dataset after using random_split in torch?
Thank you.

Output:
The paths (and labels) are stored in dataset.imgs. For instance, for imagenet:
In [ ]: print(dataset.imgs[0])
Out [ ]: ('/shareDB/imagenet/val/n01440764/ILSVRC2012_val_00000293.JPEG', 0)

After splitting the dataset, each split points to the original dataset:
In [ ]: len(train_dataset.dataset), len(valid_dataset.dataset)
Out [ ]: (50000, 50000)

However, each split also holds the indices of samples from the original dataset selected for the split. You can use these indices and the original dataset to get a list of the images selected for each split:
valid_imgs = [valid_dataset.dataset.imgs[i_] for i_ in valid_dataset.indices]
train_imgs = [train_dataset.dataset.imgs[i_] for i_ in train_dataset.indices]