Instruction: Poetry hangs when installing torch

Input:
I'm trying to add pytorch_pretrained_bert package, but it hangs on downloading torch. I've been waiting for almost 30 mins already. I'm running this command: poetry add pytorch_pretrained_bert -vvv and the output is as such:
PS C:\Users\aaaa\Desktop\AI\nexus\ocr> poetry add pytorch_pretrained_bert -vvv
Using virtualenv: C:\Users\aaaa\AppData\Local\pypoetry\Cache\virtualenvs\ocr-MxNkBiZL-py3.8
PyPI: 10 packages found for pytorch-pretrained-bert *
Using version ^0.6.2 for pytorch-pretrained-bert

Updating dependencies
Resolving dependencies...
   1: fact: ocr is 0.1.0
   1: derived: ocr
   1: fact: ocr depends on click (^8.0.1)
   1: fact: ocr depends on pytesseract (^0.3.8)
   1: fact: ocr depends on opencv-contrib-python (^4.5.2)
   1: fact: ocr depends on numpy (^1.21.0)
   1: fact: ocr depends on pdf2image (^1.16.0)
   1: fact: ocr depends on poppler-utils (^0.1.0)
   1: fact: ocr depends on deskew (^0.10.30)
   1: fact: ocr depends on pytorch-pretrained-bert (^0.6.2)
   1: fact: ocr depends on pytest (^5.2)
   1: fact: ocr depends on pytest (^5.2)
   1: selecting ocr (0.1.0)
   1: derived: pytest (>=5.2,<6.0)
   1: derived: pytorch-pretrained-bert (>=0.6.2,<0.7.0)
   1: derived: deskew (>=0.10.30,<0.11.0)
   1: derived: poppler-utils (>=0.1.0,<0.2.0)
   1: derived: pdf2image (>=1.16.0,<2.0.0)
   1: derived: numpy (>=1.21.0,<2.0.0)
   1: derived: opencv-contrib-python (>=4.5.2,<5.0.0)
   1: derived: pytesseract (>=0.3.8,<0.4.0)
   1: derived: click (>=8.0.1,<9.0.0)
   1: fact: pytest (5.4.3) depends on py (>=1.5.0)
   1: fact: pytest (5.4.3) depends on packaging (*)
   1: fact: pytest (5.4.3) depends on attrs (>=17.4.0)
   1: fact: pytest (5.4.3) depends on more-itertools (>=4.0.0)
   1: fact: pytest (5.4.3) depends on pluggy (>=0.12,<1.0)
   1: fact: pytest (5.4.3) depends on wcwidth (*)
   1: fact: pytest (5.4.3) depends on atomicwrites (>=1.0)
   1: fact: pytest (5.4.3) depends on colorama (*)
   1: selecting pytest (5.4.3)
   1: derived: colorama
   1: derived: atomicwrites (>=1.0)
   1: derived: wcwidth
   1: derived: pluggy (>=0.12,<1.0)
   1: derived: more-itertools (>=4.0.0)
   1: derived: attrs (>=17.4.0)
   1: derived: packaging
   1: derived: py (>=1.5.0)
PyPI: 1 packages found for pytorch-pretrained-bert >=0.6.2,<0.7.0
   1: fact: pytorch-pretrained-bert (0.6.2) depends on torch (>=0.4.1)
   1: fact: pytorch-pretrained-bert (0.6.2) depends on numpy (*)
   1: fact: pytorch-pretrained-bert (0.6.2) depends on boto3 (*)
   1: fact: pytorch-pretrained-bert (0.6.2) depends on requests (*)
   1: fact: pytorch-pretrained-bert (0.6.2) depends on tqdm (*)
   1: fact: pytorch-pretrained-bert (0.6.2) depends on regex (*)
   1: selecting pytorch-pretrained-bert (0.6.2)
   1: derived: regex
   1: derived: tqdm
   1: derived: requests
   1: derived: boto3
   1: derived: torch (>=0.4.1)
   1: fact: deskew (0.10.30) depends on numpy (*)
   1: fact: deskew (0.10.30) depends on scikit-image (!=0.15.0)
   1: selecting deskew (0.10.30)
   1: derived: scikit-image (!=0.15.0)
   1: fact: poppler-utils (0.1.0) depends on Click (>=7.0)
   1: selecting poppler-utils (0.1.0)
   1: fact: pdf2image (1.16.0) depends on pillow (*)
   1: selecting pdf2image (1.16.0)
   1: derived: pillow
   1: selecting numpy (1.21.0)
   1: fact: opencv-contrib-python (4.5.2.54) depends on numpy (>=1.13.3)
   1: selecting opencv-contrib-python (4.5.2.54)
   1: fact: pytesseract (0.3.8) depends on Pillow (*)
   1: selecting pytesseract (0.3.8)
   1: fact: click (8.0.1) depends on colorama (*)
   1: selecting click (8.0.1)
   1: selecting wcwidth (0.2.5)
   1: selecting pluggy (0.13.1)
   1: selecting more-itertools (8.8.0)
   1: selecting attrs (21.2.0)
   1: fact: packaging (21.0) depends on pyparsing (>=2.0.2)
   1: selecting packaging (21.0)
   1: derived: pyparsing (>=2.0.2)
   1: selecting py (1.10.0)
   1: selecting regex (2021.7.6)
   1: fact: tqdm (4.61.2) depends on colorama (*)
   1: selecting tqdm (4.61.2)
   1: fact: requests (2.25.1) depends on chardet (>=3.0.2,<5)
   1: fact: requests (2.25.1) depends on idna (>=2.5,<3)
   1: fact: requests (2.25.1) depends on urllib3 (>=1.21.1,<1.27)
   1: fact: requests (2.25.1) depends on certifi (>=2017.4.17)
   1: selecting requests (2.25.1)
   1: derived: certifi (>=2017.4.17)
   1: derived: urllib3 (>=1.21.1,<1.27)
   1: derived: idna (>=2.5,<3)
   1: derived: chardet (>=3.0.2,<5)
   1: fact: boto3 (1.17.105) depends on botocore (>=1.20.105,<1.21.0)
   1: fact: boto3 (1.17.105) depends on jmespath (>=0.7.1,<1.0.0)
   1: fact: boto3 (1.17.105) depends on s3transfer (>=0.4.0,<0.5.0)
   1: selecting boto3 (1.17.105)
   1: derived: s3transfer (>=0.4.0,<0.5.0)
   1: derived: jmespath (>=0.7.1,<1.0.0)
   1: derived: botocore (>=1.20.105,<1.21.0)
   1: fact: torch (1.9.0) depends on typing-extensions (*)
   1: selecting torch (1.9.0)
   1: derived: typing-extensions
   1: fact: scikit-image (0.18.2) depends on numpy (>=1.16.5)
   1: fact: scikit-image (0.18.2) depends on scipy (>=1.0.1)
   1: fact: scikit-image (0.18.2) depends on matplotlib (>=2.0.0,<3.0.0 || >3.0.0)
   1: fact: scikit-image (0.18.2) depends on networkx (>=2.0)
   1: fact: scikit-image (0.18.2) depends on pillow (>=4.3.0,<7.1.0 || >7.1.0,<7.1.1 || >7.1.1)
   1: fact: scikit-image (0.18.2) depends on imageio (>=2.3.0)
   1: fact: scikit-image (0.18.2) depends on tifffile (>=2019.7.26)
   1: fact: scikit-image (0.18.2) depends on PyWavelets (>=1.1.1)
   1: selecting scikit-image (0.18.2)
   1: derived: PyWavelets (>=1.1.1)
   1: derived: tifffile (>=2019.7.26)
   1: derived: imageio (>=2.3.0)
   1: derived: pillow (>=4.3.0,!=7.1.0,!=7.1.1)
   1: derived: networkx (>=2.0)
   1: derived: matplotlib (>=2.0.0,!=3.0.0)
   1: derived: scipy (>=1.0.1)
   1: selecting pillow (8.3.0)
   1: selecting pyparsing (2.4.7)
   1: selecting certifi (2021.5.30)
   1: selecting urllib3 (1.26.6)
   1: selecting idna (2.10)
   1: selecting chardet (4.0.0)
   1: fact: s3transfer (0.4.2) depends on botocore (>=1.12.36,<2.0a.0)
   1: selecting s3transfer (0.4.2)
   1: selecting jmespath (0.10.0)
   1: fact: botocore (1.20.105) depends on jmespath (>=0.7.1,<1.0.0)
   1: fact: botocore (1.20.105) depends on python-dateutil (>=2.1,<3.0.0)
   1: fact: botocore (1.20.105) depends on urllib3 (>=1.25.4,<1.27)
   1: selecting botocore (1.20.105)
   1: derived: python-dateutil (>=2.1,<3.0.0)
   1: selecting typing-extensions (3.10.0.0)
   1: fact: pywavelets (1.1.1) depends on numpy (>=1.13.3)
   1: selecting pywavelets (1.1.1)
   1: fact: tifffile (2021.7.2) depends on numpy (>=1.15.1)
   1: selecting tifffile (2021.7.2)
   1: fact: imageio (2.9.0) depends on numpy (*)
   1: fact: imageio (2.9.0) depends on pillow (*)
   1: selecting imageio (2.9.0)
   1: fact: networkx (2.5.1) depends on decorator (>=4.3,<5)
   1: selecting networkx (2.5.1)
   1: derived: decorator (>=4.3,<5)
   1: fact: matplotlib (3.4.2) depends on cycler (>=0.10)
   1: fact: matplotlib (3.4.2) depends on kiwisolver (>=1.0.1)
   1: fact: matplotlib (3.4.2) depends on numpy (>=1.16)
   1: fact: matplotlib (3.4.2) depends on pillow (>=6.2.0)
   1: fact: matplotlib (3.4.2) depends on pyparsing (>=2.2.1)
   1: fact: matplotlib (3.4.2) depends on python-dateutil (>=2.7)
   1: selecting matplotlib (3.4.2)
   1: derived: python-dateutil (>=2.7)
   1: derived: kiwisolver (>=1.0.1)
   1: derived: cycler (>=0.10)
   1: fact: scipy (1.6.1) depends on numpy (>=1.16.5)
   1: selecting scipy (1.6.1)
   1: fact: python-dateutil (2.8.1) depends on six (>=1.5)
   1: selecting python-dateutil (2.8.1)
   1: derived: six (>=1.5)
   1: selecting decorator (4.4.2)
   1: selecting kiwisolver (1.3.1)
   1: fact: cycler (0.10.0) depends on six (*)
   1: selecting cycler (0.10.0)
   1: selecting six (1.16.0)
   1: selecting colorama (0.4.4)
   1: selecting atomicwrites (1.4.0)
   1: Version solving took 0.246 seconds.
   1: Tried 1 solutions.

Finding the necessary packages for the current system

Package operations: 2 installs, 0 updates, 0 removals, 42 skipped

  • Installing six (1.16.0): Skipped for the following reason: Already installed
  • Installing jmespath (0.10.0): Skipped for the following reason: Already installed
  • Installing python-dateutil (2.8.1): Skipped for the following reason: Already installed
  • Installing urllib3 (1.26.6): Skipped for the following reason: Already installed
  • Installing botocore (1.20.105): Skipped for the following reason: Already installed
  • Installing cycler (0.10.0): Skipped for the following reason: Already installed
  • Installing kiwisolver (1.3.1): Skipped for the following reason: Already installed
  • Installing pillow (8.3.0): Skipped for the following reason: Already installed
  • Installing decorator (4.4.2): Skipped for the following reason: Already installed
  • Installing numpy (1.21.0): Skipped for the following reason: Already installed
  • Installing pyparsing (2.4.7): Skipped for the following reason: Already installed
  • Installing certifi (2021.5.30): Skipped for the following reason: Already installed
  • Installing chardet (4.0.0): Skipped for the following reason: Already installed
  • Installing imageio (2.9.0): Skipped for the following reason: Already installed
  • Installing networkx (2.5.1): Skipped for the following reason: Already installed
  • Installing colorama (0.4.4): Skipped for the following reason: Already installed
  • Installing scipy (1.6.1): Skipped for the following reason: Already installed
  • Installing pywavelets (1.1.1): Skipped for the following reason: Already installed
  • Installing matplotlib (3.4.2): Skipped for the following reason: Already installed
  • Installing tifffile (2021.7.2): Skipped for the following reason: Already installed
  • Installing typing-extensions (3.10.0.0): Skipped for the following reason: Already installed
  • Installing s3transfer (0.4.2): Skipped for the following reason: Already installed
  • Installing idna (2.10): Skipped for the following reason: Already installed
  • Installing atomicwrites (1.4.0): Skipped for the following reason: Already installed
  • Installing attrs (21.2.0): Skipped for the following reason: Already installed
  • Installing click (8.0.1): Skipped for the following reason: Already installed
  • Installing packaging (21.0): Skipped for the following reason: Already installed
  • Installing more-itertools (8.8.0): Skipped for the following reason: Already installed
  • Installing boto3 (1.17.105): Skipped for the following reason: Already installed
  • Installing py (1.10.0): Skipped for the following reason: Already installed
  • Installing requests (2.25.1): Skipped for the following reason: Already installed
  • Installing scikit-image (0.18.2): Skipped for the following reason: Already installed
  • Installing wcwidth (0.2.5): Skipped for the following reason: Already installed
  • Installing tqdm (4.61.2): Skipped for the following reason: Already installed
  • Installing torch (1.9.0)
  • Installing pluggy (0.13.1): Skipped for the following reason: Already installed
  • Installing regex (2021.7.6): Skipped for the following reason: Already installed

As you can see installation had not finished. What could be a reason? I'm using Windows 10 operating system and running the command in Windows PowerShell.

Output:
TL;DR: This is easily verified by either a green colored version number and dot as shown in this screenshot or running poetry run <package> --version and having successful output.
Short Answer
When I've installed packages with Poetry, I've seen some packages, like pylint, only display the line
Installing pylint (2.11.1)

and appear to never complete. But if I run
poetry run pylint --version

it'll print out the installed version within the poetry venv:
pylint 2.11.1
astroid 2.8.3
Python 3.9.7 (default, Sep 3 2021, ...)

Longer Answer
Not sure if this is a recent change with Poetry, but currently using the preview version of Poetry, 1.2.0a2, the output after running poetry add <package> -vvv looks something like this to indicate separately resolving dependencies and install progress:
$ poetry add pylint -vvv
Loading configuration file ...
Using virtualenv: ...
PyPI: 1 packages found for pylint

Updating dependencies
Resolving dependencies...
   1: fact: pylint (2.11.1) depends on platformdirs (>=2.2.0)
   1: fact: pylint (2.11.1) depends on astroid (>=2.8.0,<2.9)
   1: fact: pylint (2.11.1) depends on isort (>=4.2.5,<6)
   1: fact: pylint (2.11.1) depends on mccabe (>=0.6,<0.7)
   1: fact: pylint (2.11.1) depends on toml (>=0.7.1)
   1: fact: pylint (2.11.1) depends on typing-extensions (>=3.10.0)
   1: fact: pylint (2.11.1) depends on colorama (*)
   1: selecting pylint (2.11.1)
   1: derived: typing-extensions (>=3.10.0)
   1: derived: toml (>=0.7.1)
   1: derived: mccabe (>=0.6,<0.7)
   1: derived: isort (>=4.2.5,<6)
   1: derived: astroid (>=2.8.0,<2.9)
   1: derived: platformdirs (>=2.2.0)
PyPI: 1 packages found for pylint
...

Writing lock file

Finding the necessary packages for the current system

Package operations: 6 installs, 0 updates, 0 removals, 0 skipped

  • Installing pylint (2.11.1): Pending...
  • Installing pytest (2.11.1): Installing...
  • Installing pytest (2.11.1)  # dot and version will be green to indicate package was installed

Note that the output will initially start as Pending... and may update the line in place to Downloading X% and then Installing... or output on new lines. So if you see a line by itself, it is possible the package has been installed. This is easily verified by either a green colored version number and dot as shown in this screenshot or running poetry run <package> --version and having successful output.
Hopefully, this extra output will clarify the install state of the package(s) you were installing.


Instruction: Conv2D padding in TensorFlow and PyTorch

Input:
I am trying to convert TensorFlow model to PyTorch but having trouble with padding. My code for for relevant platforms are as follow:
TensorFlow
conv1 = tf.layers.conv2d(
                inputs=input_layer,
                filters=32,
                kernel_size=[5, 5],
                padding="same",
                activation=tf.nn.relu,
                name = "conv1")

PyTorch
conv1 = nn.Conv2d(1, 32, kernel_size=5, stride=1, padding=2)

I have few questions:

Are the above codes equivalent?
How many paddings are added on left/right/top/bottom if we use same padding in tensorflow?
How many paddings are added on left/right/top/bottom if we use padding=2 in pytorch?
If the above two code snippets are not equivalent then how can we make the same conv layer?

Thanks in advance.

Output:
To answer your questions:
The reason why Pytorch doesn't have padding = 'same' to quite simply put it is due to its dynamic computation graph in comparison to Tensorflow static graph.

Both the codes are not equivalent as different padding is used.

'Same' padding tries to pad evenly on the left and right, but if the amount of columns to be added is odd, it will then add an extra column to the right.

'Padding = 2' in Pytorch applies 2 implicit paddings on either side.

Pytorch 1.9 has added padding = 'same' for un-strided or stride = 1 convolutions. Which will work for your use case.

But for stride > 2 padding needs to be added manually.
Here is the good implementation to perform 'same' padding:-
https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/layers/padding.py#L28


Instruction: Why does torch.scatter requires a smaller shape for indices than values?

Input:
A similar question was already asked here, but I think the solution is not suited for my case.
I just wonder why it is not possible to do a torch.scatter operation, where my index tensor is bigger than my value tensor. In my case I have duplicate indices, e.g. the following value tensor a and the index tensor idx:
a = torch.tensor([[0, 1, 0, 0],
                  [0, 0, 1, 0]])

idx = torch.tensor([[1, 1, 2, 3, 3],
                    [0, 0, 1, 2, 2]])

a.scatter(-1, idx, 1) returns:

RuntimeError: Expected index [2, 5] to be smaller than self [2, 4] apart from dimension 1 and to be smaller size than src [2, 4]

Is there another way to achieve this?

Output:
Not a solution, but a workaround:
a = torch.tensor([[0, 1, 0, 0],
                  [0, 0, 1, 0]])

idx = torch.tensor([[1, 1, 2, 3, 3],
                    [0, 0, 1, 2, 2]])

rows = torch.arange(0, a.size(0))[:,None]
n_col = idx.size(1)
a[rows.repeat(1, n_col), idx] = 1

rows.repeat(1, n_col) gives the row index to the corresponding column index in idx.


Instruction: ValueError: no gopen handler defined

Input:
I am new to using webdataset library from pytorch. I have created .tar files of a sample dataset present locally in my system using webdataset.TarWriter(). The .tar files creation seems to be successful as I could extract them separately on windows platform and verify the same dataset files.
Now, I create train_dataset = wds.Dataset(url) where url is the local file path of the .tar files. after this, I perform the following operations:
train_loader = torch.utils.data.DataLoader(train_dataset, num_workers=0, batch_size=10)
sample = next(iter(train_loader))
print(sample)

It is resulting me in error like this
The same code works fine if I use a web url example: "http://storage.googleapis.com/nvdata-openimages/openimages-train-000000.tar" mentioned in webdataset documentation: https://reposhub.com/python/deep-learning/tmbdev-webdataset.html
I couldn't understand the error so far. Any idea on how to solve this problem?

Output:
I have had the same error since yesterday, I finally found the culprit. WebDataset/tarIterators.py makes use of WebDataset/gopen.py. In gopen.py urllib.parse.urlparse is called to parse the url to be opened, in your case the url is D:/PhD/....
gopen_schemes = dict(
    __default__=gopen_error,
    pipe=gopen_pipe,
    http=gopen_curl,
    https=gopen_curl,
    sftp=gopen_curl,
    ftps=gopen_curl,
    scp=gopen_curl)

def gopen(url, mode="rb", bufsize=8192, **kw):
    """Open the URL. This uses the `gopen_schemes` dispatch table to dispatch based on scheme.
    Support for the following schemes is built-in: pipe, file, http, https, sftp, ftps, scp.
    When no scheme is given the url is treated as a file. You can use the OPEN_VERBOSE argument to get info about files being opened.
    :param url: the source URL
    :param mode: the mode ("rb", "r")
    :param bufsize: the buffer size
    """
    global fallback_gopen
    verbose = int(os.environ.get("GOPEN_VERBOSE", 0))
    if verbose:
        print("GOPEN", url, info, file=sys.stderr)
    assert mode in ["rb", "wb"], mode
    if url == "-":
        if mode == "rb":
            return sys.stdin.buffer
        elif mode == "wb":
            return sys.stdout.buffer
        else:
            raise ValueError(f"unknown mode {mode}")
    pr = urlparse(url)
    if pr.scheme == "":
        bufsize = int(os.environ.get("GOPEN_BUFFER", -1))
        return open(url, mode, buffering=bufsize)
    if pr.scheme == "file":
        bufsize = int(os.environ.get("GOPEN_BUFFER", -1))
        return open(pr.path, mode, buffering=bufsize)
    handler = gopen_schemes["__default__"]
    handler = gopen_schemes.get(pr.scheme, handler)
    return handler(url, mode, bufsize, **kw)

As you can see in the dictionary the __default__ function is gopen_error. This is the function returning the error you are seeing. pr = urlparse(url) on your url will generate an urlparse where the scheme (pr.scheme) is 'd' because your disk is named D. However, it should be 'file' for the function to work as intended. Since it is not equal to 'file' or any of the other schemes in the dictionary (http, https, sftp, etc), the default function will be used, which returns the error. I circumvented this issue by adding d=gopen_file to the gopen_schemes dictionary. I hope this helps you further temporarily. I will address this issue on the WebDataset GitHub page as well and keep this page updated if I get a more practical update.
Good luck!


Instruction: Convert tensor of integers to binary tensor with 1 only at that index

Input:
Is there a pain free to convert a tensor of integers to a binary tensor with 1 only at that integers index in pytorch?
e.g.
tensor([[1,3,2,6]])

would become
tensor([[0,1,0,0,0,0,0],
        [0,0,0,1,0,0,0],
        [0,0,1,0,0,0,0],
        [0,0,0,0,0,0,1]])

Output:
t = tensor([[1,3,2,6]])

rows = t.shape[1]
cols = t.max() + 1

output = torch.zeros(rows, cols) # initializes zeros array of desired dimensions
output[list(range(rows)), t.tolist()] = 1 # sets cells to 1

To clarify the last operation, you can pass in a list of the row numbers and column numbers, and it will set all those elements to the value after the equal.
So in our case we want to set the following locations to 1:
(0,1), (1,3), (2,2), (3,6)

Which we'd represent as:
output[[0,1,2,3], [1,3,2,6]] = 1

And you'll see that those lists line up with a) an increasing list up to the total row count, and b) our original tensor


Instruction: How to understand decoder_start_token_id and forced_bos_token_id in mbart?

Input:
When I want to use huggingface's pretrained models such as mbart to conduct multilingual experiments, the meaning of paramaters decoder_start_token_id and forced_bos_token_id confuse me. I find codes like:
# While generating the target text set the decoder_start_token_id to the target language id.
# The following example shows how to translate English to Romanian
# using the facebook/mbart-large-en-ro model.
from transformers import MBartForConditionalGeneration, MBartTokenizer

tokenizer = MBartTokenizer.from_pretrained("facebook/mbart-large-en-ro", src_lang="en_XX")
article = "UN Chief Says There Is No Military Solution in Syria"
inputs = tokenizer(article, return_tensors="pt")
translated_tokens = model.generate(**inputs, decoder_start_token_id=tokenizer.lang_code_to_id["ro_RO"])
tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]

and:
# To generate using the mBART-50 multilingual translation models,
# eos_token_id is used as the decoder_start_token_id and the target language id is forced as the first generated token.
# To force the target language id as the first generated token,
# pass the forced_bos_token_id parameter to the generate method.
# The following example shows how to translate between Hindi to French and Arabic to English
# using the facebook/mbart-50-large-many-to-many checkpoint.
from transformers import MBartForConditionalGeneration, MBart50TokenizerFast

article_hi = "संयुक्त राष्ट्र के प्रमुख का कहना है कि सीरिया में कोई सैन्य समाधान नहीं है"
article_ar = "الأمين العام للأمم المتحدة يقول إنه لا يوجد حل عسكري في سوريا."

model = MBartForConditionalGeneration.from_pretrained("facebook/mbart-large-50-many-to-many-mmt")
tokenizer = MBart50TokenizerFast.from_pretrained("facebook/mbart-large-50-many-to-many-mmt")

# translate Hindi to French
tokenizer.src_lang = "hi_IN"
encoded_hi = tokenizer(article_hi, return_tensors="pt")
generated_tokens = model.generate(**encoded_hi, forced_bos_token_id=tokenizer.lang_code_to_id["fr_XX"])
tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
# => "Le chef de l 'ONU affirme qu 'il n 'y a pas de solution militaire en Syria."

# translate Arabic to English
tokenizer.src_lang = "ar_AR"
encoded_ar = tokenizer(article_ar, return_tensors="pt")
generated_tokens = model.generate(**encoded_ar, forced_bos_token_id=tokenizer.lang_code_to_id["en_XX"])
tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
# => "The Secretary-General of the United Nations says there is no military solution in Syria."

While the annotation of this two paramaters are:
decoder_start_token_id (:obj:`int`, `optional`):
If an encoder-decoder model starts decoding with a different token than `bos`,
the id of that token.

forced_bos_token_id (:obj:`int`, `optional`):
The id of the token to force as the first generated token after the :obj:`decoder_start_token_id`.
Useful for multilingual models like :doc:`mBART <../model_doc/mbart>` where
the first generated token needs to be the target language token.

And for different varients of mbart, such as facebook/mbart-large-cc25 and facebook/mbart-large-50, which one should we specify to generate response of specific language?

Output:
In the standard sequence-to-sequence models, the decoding starts with providing the decoder with the [bos] symbols, it generates the word w1, which is provided as the input of the decoder in the next step. and the decoder generates the word w2. This continues until the [eos] (end-of-sentence) token is generated.
[bos] w_1  w_2  w_3
  ↓    ↓    ↓    ↓
┌──────────────────┐
│     DECODER      │
└──────────────────┘
  ↓    ↓    ↓    ↓
 w_1  w_2  w_2 [eos]

With mBART, this is more tricky because you need to tell it what the target language and source language are. For the encoder and for training data, the tokenizer takes care of that and adds the language-specific tags at the end of the source sentence and at the beginning of the target sentence. The sentences are then in format (given the source has 4 words and the target 3):

source: v1 v2 v3 v4 [src_lng]
target: [tgt_lng] w1 w2 w3 [eos]

Unlike training, at inference time, the target sentence is unknown and you want to generate it. But you still need to say the decoder what it should use instead of the generic [bos] token. This where the forced_bos_token_id comes into play. It is still the tokenizer that knows the IDs of the specific tokens. Different mBARTs have different tokenizers, you should always use the language IDs for the tokenizer that matches the models.
The attributes you mention seem to do the same thing, but I would stick to forced_bos_token_id mentioned in the mBART documentation. The method APIs in HuggingFace Transformers are very permissive and some of the attributes only apply to some models and get ignored by others. I would avoid using something that is not explicitly mentioned in the documentation of the particular model.


Instruction: Classification in LSTM returns same value for classification

Input:
This is my first time posting in stack overflow so forgive me if I do any sort of mistake.
I have 10000 data, and each data has a label of 0 and 1. I want to perform classification using LSTM as this is time series data.
input_dim = 1
hidden_dim = 32
num_layers = 2
output_dim = 1

# Here we define our model as a class
class LSTM(nn.Module):
  def __init__(self, input_dim, hidden_dim, num_layers, output_dim):
    super(LSTM, self).__init__()
    self.hidden_dim = hidden_dim
    self.num_layers = num_layers
    self.lstm = nn.LSTM(input_dim, hidden_dim, num_layers, batch_first=True)
    self.fc = nn.Linear(hidden_dim, output_dim)

  def forward(self, x):
    #Initialize hidden layer and cell state
    h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_dim).requires_grad_()
    c0 = torch.zeros(self.num_layers, x.size(0), self.hidden_dim).requires_grad_()

    # We need to detach as we are doing truncated backpropagation through time (BPTT)
    # If we don't, we'll backprop all the way to the start even after going through another batch
    out, (hn, cn) = self.lstm(x, (h0.detach(), c0.detach()))

    # Index hidden state of last time step
    # out.size() --> 100, 32, 100
    # out[:, -1, :] --> 100, 100 --> just want last time step hidden states!
    out = self.fc(out[:, -1, :])

    # For binomial Classification
    m = torch.sigmoid(out)

    return m

model = LSTM(input_dim=input_dim, hidden_dim=hidden_dim, output_dim=output_dim, num_layers=num_layers)
loss = nn.BCELoss()

optimiser = torch.optim.Adam(model.parameters(), lr=0.00001, weight_decay=0.00006)

num_epochs = 100

# Number of steps to unroll
seq_dim =look_back-1

for t in range(num_epochs):
    y_train_class = model(x_train)

    output = loss(y_train_class, y_train)

    # Zero out gradient, else they will accumulate between epochs
    optimiser.zero_grad(set_to_none=True)

    # Backward pass
    output.backward()

    # Update parameters
    optimiser.step()

This is an example of what the result looks like
This code is initially from kaggle, I edited them for classification. Please, can you tell me what I am doing wrong?
EDIT 1:
Add dataloader
from torch.utils.data import DataLoader
from torch.utils.data import TensorDataset
x_train = torch.from_numpy(x_train).type(torch.Tensor)
y_train = torch.from_numpy(y_train).type(torch.Tensor)
x_test = torch.from_numpy(x_test).type(torch.Tensor)
y_test = torch.from_numpy(y_test).type(torch.Tensor)

train_dataloader = DataLoader(TensorDataset(x_train, y_train), batch_size=128, shuffle=True)
test_dataloader = DataLoader(TensorDataset(x_test, y_test), batch_size=128, shuffle=True)

I realized I had forgotten to inverse the transformation before checking the result. When I did that, I got different values from classification, however all values are in the scale of 0.001-0.009, so when I round them, the result is same. Label 0 for all classification.

Output:
A common phenomenon in NN training is that they will initially converge to a very naive solution to the problem where they output a constant prediction that minimizes the error on the training data. My guess is that in your training data, the ratio between 0 and 1 classes is close to 0.5423. Depending on whether your model is of sufficient complexity, it might learn to make more specific predictions based on the input when given more learning steps.
While increasing the number of epochs could help, there is something better you can do with your current setup. Currently, you are only performing a single optimizer step per epoch. Typically, you would want a step per batch and loop over your data in (mini)batches of, say, 32 inputs for example. To do this, it would be best to use a DataLoader where you can define a batch size, and loop over the dataloader inside your epoch loop similar to this example.


Instruction: Hermetic / Non Hermetic Packages in Python

Input:
While going through PyTorch documentation, I came across the term hermetic packages:

torch.package adds support for creating hermetic packages containing arbitrary PyTorch code. These packages can be saved, shared, used to load and execute models at a later date or on a different machine, and can even be deployed to production using torch::deploy.

I don't understand what hermetic packages mean in this context.

Can someone explain what makes packages hermetic?
What would non-hermetic packages look like?

With some search over Stack Overflow [1][2], it seems this terminology is a generic term used in software world. Any examples - even outside of PyTorch/Python world -  would help in solidifing my understanding.
Thank you!

[1] Creating Hermetic Maven Builds
[2] Bazel: hermetic use of jar command?

Output:
In the context, hermatic is used to mean that the already preinstalled libraries and configuration of your machine you are running on (macos laptop, to windows desktop, etc.) will be able to build Pytorch and its depedancies in an identical way.
The following link has a section on hermatic builds:
https://www.google.com/search?q=what+is+hermatic+mean+in+software&oq=what+is+hermatic+mean+in+software&aqs=chrome..69i57j33l3.7998j0j7&sourceid=chrome&ie=UTF-8
"Our builds are hermetic, meaning that they are insensitive to the libraries and other software installed on the build machine. Instead, builds depend on known versions of build tools, such as compilers, and dependencies, such as libraries. The build process is self-contained and must not rely on services that are external to the build environment."
This is also a good link to refer to:
https://news.ycombinator.com/item?id=19610869
Sarthak Jain


Instruction: Retrieving intermediate features from pytorch torch.hub.load

Input:
I have a Net object instantiated in pytorch via torch.hub.load:
model = torch.hub.load('facebookresearch/pytorchvideo', 'slowfast_r50', pretrained=True)

The final layer is a projection to a 400-dim vector. Is there a way to get the pentultimate layer instead during a forward pass?

Output:
Yes, easiest way is to switch the layer with torch.nn.Identity (which simply returns it's inputs unchanged):
Line below changes this submodule:
(6): ResNetBasicHead(
      (dropout): Dropout(p=0.5, inplace=False)
      (proj): Linear(in_features=2304, out_features=400, bias=True)
      (output_pool): AdaptiveAvgPool3d(output_size=1)
    )

to Identity:
model.blocks[6] = torch.nn.Identity()

as you probably don't want to keep the Dropout anyway (you might only change proj or any other part of the network as needed).


Instruction: My Loss Function doesn't get smaller values during training

Input:
I am trying to predict the center of my palm
The structure of my neural network consists of 2 cnn which both are followed by max-pooling and a linear layer that has 2 outputs, one for x and the other one for y. The input is a 720x720 image.
class MyNeuralNetwork(torch.nn.Module):
    def __init__(self):
        super(MyNeuralNetwork, self).__init__()
        self.conv1 = torch.nn.Conv2d(4, 5, 5)
        self.conv2 = torch.nn.Conv2d(5, 5, 5)
        self.pool = torch.nn.MaxPool2d(3, 3)
        self.linear = torch.nn.Linear(5 * 78 * 78, 2)

    def forward(self, x):
        x = self.conv1(x)
        x = self.pool(x)
        x = self.conv2(x)
        x = self.pool(x)
        x = x.view(x.size(0), -1)
        x = self.linear(x)
    return x

I have the pathnames of the images saved in a csv file. the x and y coordinates are saved in a different csv file. Here is the code for my Dataset.
class MyHand(Dataset):
 """Creating the proper dataset to feed my neural network"""
    def __init__(self, name_path, root_dir, results_path, transform=None):
       self.names = pd.read_csv(name_path)
       self.rootdir = root_dir
       self.transform = transform
       self.results = pd.read_csv(results_path)

    def __len__(self):
        length = len(self.names.columns)
        return length

    def __getitem__(self, index):
        img_path = os.path.join(self.rootdir, self.names.columns[index])
        image = pl.imread(img_path)
        x_top_left_corner = torch.tensor(self.results.iloc[index, 0])
        y_top_left_corner = torch.tensor(self.results.iloc[index, 1])
        width = torch.tensor(self.results.iloc[index, 2])
        height = torch.tensor(self.results.iloc[index, 3])

        # calculating the x and y center of my palm
        x_center = x_top_left_corner + width/2
        y_center = y_top_left_corner - height/2

        if self.transform:
            image = self.transform(image)

        return image, x_center, y_center

and the code for training the network is
dataset = MyHand(name_path='path to the names of the images csv',
results_path='path to the results cvs',
transform=torchvision.transforms.ToTensor( ))
loader = DataLoader(dataset=dataset, batch_size=4)
model = MyNeuralNetwork()
criterion = torch.nn.MSELoss()
EPOCHS = 5
LEARNING_RATE = 0.001
optimizer = optim.SGD(model.parameters(), LEARNING_RATE)

for epoch in range(EPOCHS):
    print("epoch:", epoch)
    for data in dataset:
        pic, x, y = data
        model.zero_grad()
        outpout = model(pic[None, :, :, :])
        loss1 = criterion(outpout[0, 0], x)
        loss2 = criterion(outpout[0, 1], y)
        loss = loss1 + loss2
        loss.backward()
        print(loss)

but as you can see below my loss function has exactly the same results at each epoch and it doesn't decrease at all. What can i do for that? I tried different values of learning rate but still the same.

Output:
Your loss values are extremly high as you see. I would propose that you normalize your outputs by using the sigmoid activation function. Now the coordinates are in the range 0-1 and can be later translated to the image by multiplying them with 720. To calculate the loss, you have to divide your target cooridnates by 720. Then you should get a nice and stable loss in the range 0-1.
Also:

either decay your learning rate or try a smaller one
scale your image down (I don't know how the images look like but 720x720 is quite big)
use three convolutions with smaller kernels
add a second linear layer


Instruction: How to calculate Gradient of the loss with respect to input?

Input:
I have a pre-trained PyTorch model. I need to calculate the gradient of the loss with respect to the network's inputs using this model (without training again and only using the pre-trained model).
I wrote the following code, but I am not sure it is correct or not.
    test_X, test_y = load_data(mode='test')
    testset_original = MyDataset(test_X, test_y, transform=default_transform)
    testloader = DataLoader(testset_original, batch_size=32, shuffle=True)

    model = MyModel(device=device).to(device)
    checkpoint = torch.load('checkpoint.pt')
    model.load_state_dict(checkpoint['model_state_dict'])

    gradient_losses = []
    for i, data in enumerate(testloader):
        inputs, labels = data
        inputs= inputs.to(device)
        labels = labels.to(device)
        inputs.requires_grad = True
        output = model(inputs)
        loss = loss_function(output)
        loss.backward()
        gradient_losses.append(inputs.grad)

My question is, does this list gradient_losses actually storing what I wish to store? If not, what is the correct way to do that?

Output:
does this list gradient_losses actually storing what I wish to store?

Yes, if you are looking to get the derivative of the loss with respect to the input then that seems to be the correct way to do it. Here is minimal example, take f(x) = a*x. Then df/dx = a.
>>> x = torch.rand(10, requires_grad=True)
>>> y = torch.rand(10)
>>> a = torch.tensor([3.], requires_grad=True)

>>> loss = a*x - y
>>> loss.mean().backward()

>>> x.grad
tensor([0.3000, 0.3000, ..., 0.3000, 0.3000])

Which, in this case is equal to a / len(x)
Do note, each gradient you extract with input.grad will be averaged over the whole batch, and won't be a gradient over each individual input.

Also, you don't need to .clone() your input gradients as they are not part of the model and won't get zeroed by model.zero_grad().


Instruction: Saving a trained Detectron2 model and making predictions on a single image

Input:
I am new to detectron2 and this is my first project. After reading the docs and using the tutorials as a guide, I trained my model on the custom dataset and performed the evaluation.
I would now like to make predictions on images I receive via an API by loading this saved model. I could not find any reading materials that could help me with this task.
To save my model, I have used this link as a reference - https://detectron2.readthedocs.io/en/latest/tutorials/models.html
I am able to save my trained model using the following code-
from detectron2.modeling import build_model
model = build_model(cfg) # returns a torch.nn.Module

from detectron2.checkpoint import DetectionCheckpointer

checkpointer = DetectionCheckpointer(model, save_dir="output")
checkpointer.save("model_final") # save to output/model_final.pth

But I am still confused as to how I can go about implementing what I want. I could use some guidance on what my next steps should be. Would be extremely grateful to anyone who can help.

Output:
for a single image, create a list of data. Put image path in the file_name as below:
test_data = [{'file_name': '.../image_1jpg',
              'image_id': 10}]

Then do run the following:
from detectron2.config import get_cfg
from detectron2.engine import DefaultPredictor
from detectron2.data import MetadataCatalog
from detectron2.utils.visualizer import Visualizer, ColorMode
import matplotlib.pyplot as plt
import cv2.cv2 as cv2

test_data = [{'file_name': '.../image_1jpg',
                  'image_id': 10}]

cfg = get_cfg()
cfg.merge_from_file("model config")
cfg.MODEL.WEIGHTS = "model_final.pth" # path for final model
predictor = DefaultPredictor(cfg)
im = cv2.imread(test_data[0]["file_name"])
outputs = predictor(im)
v = Visualizer(im[:, :, ::-1],
               metadata=MetadataCatalog.get(cfg.DATASETS.TRAIN[0]),
               scale=0.5,
               instance_mode=ColorMode.IMAGE_BW)
out = v.draw_instance_predictions(outputs["instances"].to("cpu"))
img = cv2.cvtColor(out.get_image()[:, :, ::-1], cv2.COLOR_RGBA2RGB)
plt.imshow(img)

This will show the prediction for the single image


Instruction: module 'torch' has no attribute 'nan_to_num'

Input:
I am using 1.7.1 version of Pytorch on Ubuntu, and I try to do the following :
x = torch.tensor([float('nan'), float('inf'), -float('inf'), 3.14])
torch.nan_to_num(x)

but I am getting this error :
AttributeError: module 'torch' has no attribute 'nan_to_num'
But it does exist in the documentation since I just copied those 2 lines from it. Can someone help me ?

Output:
nan_to_num was introduced in PyTorch 1.8. You will need to update your torch package to access it:
pip install --upgrade torch


Instruction: pytorch loss function for regression model with a vector of values

Input:
I'm training a CNN architecture to solve a regression problem using PyTorch where my output is a tensor of 25 values. The input/target tensor could be either all zeros or a gaussian distribution with a sigma value of 2. An example of a 4-sample batch is as this one:
[[0.13534, 0.32465, 0.60653, 0.8825, 1.0000, 0.88250,0.60653, 0.32465, 0.13534, 0.043937, 0.011109, 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,  0., 0.],
 [0., 0., 0.,  0., 0., 0., 0.13534, 0.32465, 0.60653, 0.8825, 1.0000, 0.88250,0.60653, 0.32465, 0.13534, 0.043937, 0.011109, 0., 0., 0., 0., 0., 0., 0., 0.],
  [0., 0., 0.,  0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.13534, 0.32465, 0.60653, 0.8825, 1.0000, 0.88250,0.60653, 0.32465, 0.13534 ],
  [0., 0., 0.,  0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]]

My question is how to design a loss function for the model effectively learn the regression output with 25 values.
I have tried 2 types of loss, torch.nn.MSELoss() and torch.nn.MSELoss()-torch.nn.CosineSimilarity(). They sort of work. However, sometimes the network has difficulty converging, especially when there are a lot of samples with all "zeros", which leads the network to output a vector with all 25 small values.
My question is, is there any other loss which we could try?

Output:
Your values do not seem widely different in scale so an MSELoss seems like it would work fine. Your model could be collapsing because of the many zeros in your target.
You can always try torch.nn.L1Loss() (but I do not expect it to be much better than torch.nn.MSELoss())
I suggest that you instead try to predict the gaussian mean/mu, and later try to re-create the gaussian for each sample if you really need it.
So you have two alternatives if you choose to try this method.
Alt 1
A good alternative is to encode your target to look like a classification target. Your 25 element vectors become a single value where the original target == 1 (possible classes will 0, 1, 2, ..., 24). We can then assign a sample that contains "only zeroes" as our last class "25". So your target:
[[0.13534, 0.32465, 0.60653, 0.8825, 1.0000, 0.88250,0.60653, 0.32465, 0.13534, 0.043937, 0.011109, 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,  0., 0.],
 [0., 0., 0.,  0., 0., 0., 0.13534, 0.32465, 0.60653, 0.8825, 1.0000, 0.88250,0.60653, 0.32465, 0.13534, 0.043937, 0.011109, 0., 0., 0., 0., 0., 0., 0., 0.],
  [0., 0., 0.,  0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.13534, 0.32465, 0.60653, 0.8825, 1.0000, 0.88250,0.60653, 0.32465, 0.13534 ],
  [0., 0., 0.,  0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]]

becomes
[4,
10,
20,
25]

If you do this, then you can try the common torch.nn.CrossEntropyLoss().
I do not know what your dataloader looks like but given a single sample in your original format, you can convert it to my proposed format with:
def encode(tensor):
    if tensor.sum() == 0:
        return len(tensor)
    return torch.argmax(tensor)

and back to a gaussian with:
def decode(value):
    n_values = 25
    zero = torch.zeros(n_values)
    if value == n_values:
        return zero
    # Create gaussian around value
    std = 2
    n = torch.arange(n_values) - value
    sig = 2*std**2
    gauss = torch.exp(-n**2 / sig2)
    # Only return 9 values from the gaussian
    start_ix = max(value-6, 0)
    end_ix = min(value+7,n_values)
    zero[start_ix:end_ix] = gauss[start_ix:end_ix]
    return zero

(Note I have not tried them with batches, only samples)
Alt 2
The second option is to change your regression targets (still only the argmax positions (mu)) to a nicer regression value in the range 0-1 and have a separate neuron that outputs a "mask value" (also 0-1). Then your batch of:
[[0.13534, 0.32465, 0.60653, 0.8825, 1.0000, 0.88250,0.60653, 0.32465, 0.13534, 0.043937, 0.011109, 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,  0., 0.],
 [0., 0., 0.,  0., 0., 0., 0.13534, 0.32465, 0.60653, 0.8825, 1.0000, 0.88250,0.60653, 0.32465, 0.13534, 0.043937, 0.011109, 0., 0., 0., 0., 0., 0., 0., 0.],
  [0., 0., 0.,  0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.13534, 0.32465, 0.60653, 0.8825, 1.0000, 0.88250,0.60653, 0.32465, 0.13534 ],
  [0., 0., 0.,  0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]]

becomes
# [Mask, mu]
[
[1, 0.1666], # True, 4/24
[1, 0.4166], # True, 10/24
[1, 0.8333], # True, 20/24
[0, 0]       # False, undefined
]

If you are using this setup, then you should be able to use an MSELoss with modification:
def custom_loss(input, target):
    # Assume target and input is of shape [Batch, 2]
    mask = target[...,1]
    mask_loss = torch.nn.functional.mse_loss(input[...,0], target[...,0])
    mu_loss = torch.nn.functional.mse_loss(mask*input[...,1], mask*target[...,1])
    return (mask_loss + mu_loss) / 2

This loss would only look at the 2nd value (mu) if the mask of the target is 1. Otherwise it only tried to optimize for the correct mask.
To encode to this format you would use:
def encode(tensor):
    n_values = 25
    if tensor.sum() == 0:
        return torch.tensor([0,0])
    return torch.argmax(tensor) / (n_values-1)

and to decode:
def decode(tensor):
    n_values = 25

    # Parse values
    mask, value = tensor
    mask = torch.round(mask)
    value = torch.round((n_values-1)*value)

    zero = torch.zeros(n_values)
    if mask == 0:
        return zero
    # Create gaussian around value
    std = 2
    n = torch.arange(n_values) - value
    sig = 2*std**2
    gauss = torch.exp(-n**2 / sig2)
    # Only return 9 values from the gaussian
    start_ix = max(value-6, 0)
    end_ix = min(value+7,n_values)
    zero[start_ix:end_ix] = gauss[start_ix:end_ix]
    return zero


Instruction: First argument error in pytorch.loads() function when working with Emotic demo

Input:
Basically, I'm creating an emotion recognition application, and I'm using Emotic's image dataset. They have their own premade program and trained model for a demo (The colab link below) but for some reason the third cell under  I. Prepare places pretrained model is encountering the error:
the first argument must be callable on line 19 in the 4th Google Colab cell.
Code:
# Converting model weights to python3.6 format
import torch
from PIL import Image
from torch.autograd import Variable as V
import torchvision.models as models
from torchvision import transforms as trn
from torch.nn import functional as F
import os

model_path = './places'
archs = ['resnet18']
for arch in archs:
    model_file = os.path.join(model_path,'%s_places365.pth.tar' % arch)
    save_file = os.path.join(model_path,'%s_places365_py36.pth.tar' % arch)

    from functools import partial
    import pickle
    pickle.load = partial(pickle.load, encoding="latin1")
    pickle.Unpickler = partial(pickle.Unpickler, encoding="latin1")
    model = torch.load(model_file, map_location=lambda storage, loc: storage, pickle_module=pickle)
    torch.save(model, save_file)
    print('converting %s -> %s'%(model_file, save_file))

print ('completed cell')
# Saving the model weights to use ahead in the notebook

# the architecture to use
arch = 'resnet18'
model_weight = os.path.join(model_path, 'resnet18_places365_py36.pth.tar')

# create the network architecture
model = models.__dict__[arch](num_classes=365)

#model_weight = '%s_places365.pth.tar' % arch

checkpoint = torch.load(model_weight) # model trained in GPU could be deployed in CPU machine like this!
state_dict = {str.replace(k,'module.',''): v for k,v in checkpoint['state_dict'].items()} # the data parallel layer will add 'module' before each layer name
model.load_state_dict(state_dict)
model.eval()

model.cpu()
torch.save(model, os.path.join(model_path, 'res_context' + '.pth'))
print ('completed cell')

Does anyone have any idea why this is happening? (Haven't changed any code this is the demo offered by Emotic)
Error:
TypeError                                 Traceback (most recent call last)
<ipython-input-8-1a9e3bc55eae> in <module>()
     17 #model_weight = '%s_places365.pth.tar' % arch
     18
---> 19 checkpoint = torch.load(model_weight) # model trained in GPU could be deployed in CPU machine like this!
     20 state_dict = {str.replace(k,'module.',''): v for k,v in checkpoint['state_dict'].items()} # the data parallel layer will add 'module' before each layer name
     21 model.load_state_dict(state_dict)

1 frames
/usr/local/lib/python3.7/dist-packages/torch/serialization.py in _load(zip_file, map_location, pickle_module, pickle_file, **pickle_load_args)
    867     # because it's marked readonly in pickle.
    868     # The type: ignore is because mypy can't statically determine the type of this class.
--> 869     class UnpicklerWrapper(pickle_module.Unpickler):  # type: ignore[name-defined]
    870         # from https://stackoverflow.com/questions/13398462/unpickling-python-objects-with-a-changed-module-path/13405732
    871         # Lets us override the imports that pickle uses when unpickling an object.

TypeError: the first argument must be callable

Public Gopogle Colab link:
https://colab.research.google.com/github/Tandon-A/emotic/blob/master/Colab_train_emotic.ipynb

Output:
I am the author of this repository. I had fixed this issue back in August 2021. The issue is caused due to some changes in the Python Pickle module in the python 3.7 version. The code used to work properly in the Python 3.6 version.
You can try the Colab_train_emotic.ipynb file.


Instruction: Facing issue: UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 918: ordinal not in range(128) while trying to load a Pytorch model

Input:
I am trying to load a pre-trained Pytorch model but getting an error as shown below:
model = torch.load('a.pth')
  File "/home/ubuntu/env/lib/python3.6/site-packages/torch/serialization.py", line 267, in load
    return _load(f, map_location, pickle_module)
  File "/home/ubuntu/env/lib/python3.6/site-packages/torch/serialization.py", line 420, in _load
    result = unpickler.load()
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 918: ordinal not in range(128)

I am using Python 3.6 in the virtual environment and PyTorch version = 0.3.1, any leads towards resolving the issue will be helpful.

Output:
From documentation:

By default, we decode byte strings as utf-8. This is to avoid a common error case UnicodeDecodeError: 'ascii' codec can't decode byte 0x... when loading files saved by Python 2 in Python 3. If this default is incorrect, you may use an extra encoding keyword argument to specify how these objects should be loaded, e.g., encoding='latin1' decodes them to strings using latin1 encoding, and encoding='bytes' keeps them as byte arrays which can be decoded later with byte_array.decode(...).

Try to change encoding, for instance:
model = torch.load('a.pth', encoding='latin')  # or 'ascii'


Instruction: Are there any difference between Y and *Y, where Y is a list used as input argument?

Input:
I was using the torch.tensor.repeat()
x = torch.tensor([[1, 2, 3], [4, 5, 6]])
period = x.size(1)
repeats = [1,2]
result = x.repeat(*repeats)

the result is
tensor([[1, 2, 3, 1, 2, 3],
    [4, 5, 6, 4, 5, 6]])

if I get result as follows
result = x.repeat(repeats)

the result is the same
tensor([[1, 2, 3, 1, 2, 3],
    [4, 5, 6, 4, 5, 6]])

It seems that x.repeat(repeats) and x.repeat(*repeats) work the same.
Does it mean that, for an input parameter, e.g, Y, I can use either Y or *Y

Output:
Kinda. If repeats is a (list or tuple) of ints, then it is equivalent. But in general the rule appears to be:

If the first argument is a list or tuple, take that as repeats. Ignore all other arguments.
Otherwise, take the full *args as repeats

So if your repeats is something weird like repeats=((1,2),(3,4)), then a.repeat(*repeats) succeeds and is the same as a.repeat(1, 2)==a.repeat((1, 2)) and a.repeat(repeats) fails.
Note: This is observational based on my own tests. The only official documentation is the defined type of the function, e.g. repeat(torch.Size or int...) which isn't perfectly clear with regards to semantics.

You can also get error messages like this:
TypeError: repeat(): argument 'repeats' (position 1) must be tuple of ints, not tuple

when you pass floats. In general error reporting could be better.


Instruction: Plot loss and accuracy over each epoch for both training and test datasets

Input:
I am training that model to classify 3 classes (0,1,2). I am using cross validation for 2 fold, I am using pytorch, I would like to plot the accuracy and loss function for training and test dataset over the number epochs on the same plot. I do know how to do that . especially I just evaluate the test once I finish training , Is there is way that I can have that plot for both training  data and test data
# Configuration options
k_folds = 2
loss_function = nn.CrossEntropyLoss()
# For fold results
results = {}
# Set fixed random number seed
torch.manual_seed(42)
# Prepare dataset by concatenating Train/Test part; we split later.
training_set = CustomDataset('one_hot_train_data.txt','train_3states_target.txt') #training_set = CustomDataset_3('one_hot_train_data.txt','train_5_target.txt')
training_generator = torch.utils.data.DataLoader(training_set, **params)
val_set = CustomDataset('one_hot_val_data.txt','val_3states_target.txt')
test_set = CustomDataset('one_hot_test_data.txt','test_3states_target.txt')
#testloader = torch.utils.data.DataLoader(test_set, **params)
#dataset1 = ConcatDataset([training_set, val_set])
dataset = ConcatDataset([training_set,test_set])
kfold = KFold(n_splits=k_folds, shuffle=True)

# Start print
print('--------------------------------')
# K-fold Cross Validation model evaluation
for fold, (train_ids, test_ids) in enumerate(kfold.split(dataset)):
    # Print
    print(f'FOLD {fold}')
    print('--------------------------------')
    # Sample elements randomly from a given list of ids, no replacement.
    train_subsampler = torch.utils.data.SubsetRandomSampler(train_ids)
    test_subsampler = torch.utils.data.SubsetRandomSampler(test_ids)

    # Define data loaders for training and testing data in this fold
    trainloader = torch.utils.data.DataLoader(
        dataset,**params, sampler=train_subsampler)
    testloader = torch.utils.data.DataLoader(
       dataset,
       **params, sampler=test_subsampler)
    # Init the neural network
    model = PPS()
    model.to(device)
    # Initialize optimizer
    optimizer = optim.SGD(model.parameters(), lr=LEARNING_RATE)
    # Run the training loop for defined number of epochs
    train_acc = []
    for epoch in range(0, N_EPOCHES):
        # Print epoch
        print(f'Starting epoch {epoch + 1}')
        # Set current loss value
        running_loss = 0.0
        epoch_loss = 0.0
        a = []
        # Iterate over the DataLoader for training data
        for i, data in enumerate(trainloader, 0):
            inputs, targets = data
            inputs = inputs.unsqueeze(-1)
            #inputs = inputs.to(device)
            targets = targets.to(device)
            inputs = inputs.to(device)
            # print(inputs.shape,targets.shape)
            # Zero the gradients
            optimizer.zero_grad()
            # Perform forward pass
            loss,outputs = model(inputs,targets)
            outputs = outputs.to(device)

            # Perform backward pass
            loss.backward()
            # Perform optimization
            optimizer.step()
            # print statistics
            running_loss += loss.item()
            epoch_loss += loss
            a.append(torch.sum(outputs == targets))
            # print(outputs.shape,outputs.shape[0])

            if i % 2000 == 1999:  # print every 2000 mini-batches
                print('[%d, %5d] loss: %.3f' %
                      (epoch + 1, i + 1, running_loss / 2000), "acc",
                      torch.sum(outputs == targets) / float(outputs.shape[0]))
                running_loss = 0.0
            # sum_acc += (outputs == stat_batch.argmax(1)).float().sum()
        print("epoch", epoch + 1, "acc", sum(a) / len(train_subsampler), "loss", epoch_loss / len(trainloader))
        train_acc.append(sum(a) / len(train_subsampler))
    state = {'epoch': epoch + 1, 'state_dict': model.state_dict(),
                     'optimizer': optimizer.state_dict() }
    torch.save(state, path + name_file + "model_epoch_i_" + str(epoch) + str(fold)+".cnn")
    #torch.save(model.state_dict(), path + name_file + "model_epoch_i_" + str(epoch) + ".cnn")
    # Print about testing
    print('Starting testing')

# Evaluation for this fold
    correct, total = 0, 0
    with torch.no_grad():
    # Iterate over the test data and generate predictions
     for i, data in enumerate(testloader, 0):
        # Get inputs
        inputs, targets = data
        #targets = targets.to(device)
        inputs = inputs.unsqueeze(-1)
        inputs = inputs.to(device)
        # Generate outputs
        loss,outputs = model(inputs,targets)
        outputs.to(device)
        print("out",outputs.shape)
        print("target",targets.shape)
        print("targetsize",targets.size(0))
        print("sum",(outputs == targets).sum().item())
        #print("sum",torch.sum(outputs == targets))

        # Set total and correct
       # _, predicted = torch.max(outputs.data, 1)
        total += targets.size(0)
        correct += (outputs == targets).sum().item()
        #correct += torch.sum(outputs == targets)

    # Print accuracy
    print('Accuracy for fold %d: %d %%' % (fold,float( 100.0 * float(correct / total))))
    print('--------------------------------')
    results[fold] = 100.0 * float(correct / total)

# Print fold results
print(f'K-FOLD CROSS VALIDATION RESULTS FOR {k_folds} FOLDS')
print('--------------------------------')
sum = 0.0
for key, value in results.items():
    print(f'Fold {key}: {value} %')
    sum += value
print(f'Average: {float(sum / len(results.items()))} %')

Output:
You could use Tensorboard that is built especially for that, here is the doc for pytorch : https://pytorch.org/docs/stable/tensorboard.html
So in your case when you are printing the result, you can just do a
writer.add_scalar('accuracy/train',  torch.sum(outputs == targets) / float(outputs.shape[0]), n_iter)

EDIT : adding small example that you can follow
Let's say that you are training a model :
model_name = 'network'
log_name = '{}_{}'.format(model_name, strftime('%Y%m%d_%H%M%S'))
writer = SummaryWriter('logs/{}'.format(log_name))

net = Model()
criterion = torch.nn.MSELoss()
optimizer = torch.optim.SGD(net.parameters(), lr=0.1)

for epoch in range(num_epochs):
    losses = []
    for i, (inputs,labels) in enumerate (trainloader):
        inputs = Variable(inputs.float())
        labels = Variable(labels.float())
        outputs = net(inputs)
        optimizer.zero_grad()
        loss = criterion(outputs, labels)
        losses.append(loss)
        loss.backward()
        optimizer.step()
        correct_values += (outputs == labels).float().sum()
    accuracy = 100 * correct_values / len(training_set)
    avg_loss = sum(losses) / len(training_set)
    writer.add_scalar('loss/train', avg_loss.item(), epoch)
    writer.add_scalar('acc/train', accuracy, epoch)


Instruction: FileNotFoundError: [Errno 2] No such file or directory: '.data/multi30k/train.fr'

Input:
I'm trying to load Multi30k torchtext dataset using google colab. When I load the .de it works fine, but as soon as I changed from .de I get this error:
FileNotFoundError: [Errno 2] No such file or directory: '.data/multi30k/train.fr'

This is how I loaded the .de and it worked:
train_data, valid_data, test_data = datasets.Multi30k.splits(
    root=".data",
    exts=('.de', '.en'),
    fields = (SRC, TRG),

)

As soon as I changed this code by changing .de to .fr the error rises:
train_data, valid_data, test_data = datasets.Multi30k.splits(
    root=".data",
    exts=('.fr', '.en'),
    fields = (SRC, TRG),

)

Imports
import torch
from torch import nn
from torch.nn  import functional as F
import spacy, math, random
import numpy as np
from torchtext.legacy import datasets, data
import time
from prettytable import PrettyTable
from matplotlib import pyplot as plt

Seeds
SEED = 42
np.random.seed(SEED)
torch.manual_seed(SEED)
random.seed(SEED)
torch.cuda.manual_seed(SEED)
torch.backends.cudnn.deteministic = True

Tokenizers spacy
import spacy
spacy.cli.download('fr_core_news_sm')

spacy_fr = spacy.load('fr_core_news_sm')
spacy_en = spacy.load('en_core_web_sm')

def tokenize_fr(sent):
  return [tok.text for tok in spacy_fr.tokenizer(sent)]

def tokenize_en(sent):
  return [tok.text for tok in spacy_en.tokenizer(sent)]

Fields
SRC = data.Field(
    tokenize= tokenize_fr,
    lower= True,
    init_token = "<sos>",
    eos_token = "<eos>",
    include_lengths =True
)

TRG = data.Field(
    tokenize = tokenize_en,
    lower= True,
    init_token = "<sos>",
     eos_token = "<eos>"
)

The cell that throws an error
train_data, valid_data, test_data = datasets.Multi30k.splits(
    root=".data",
    exts=('.fr', '.en'),
    fields = (SRC, TRG),
)

Output:
It's because there is no train.fr file in the dataset itself.
If you list down what pytorch downloaded,
$ !ls -al .data/multi30k
total 5.4M
drwxr-xr-x 2 root root 4.0K Jul 15 14:26 .
drwxr-xr-x 3 root root 4.0K Jul 15 14:26 ..
-rw-r--r-- 1 root root  65K Jul 15 14:26 mmt_task1_test2016.tar.gz
-rw-rw-r-- 1 1000 1000  69K Oct 17  2016 test2016.de
-rw-rw-r-- 1 1000 1000  61K Oct 17  2016 test2016.en
-rw-rw-r-- 1 1000 1000  71K Feb 11  2017 test2016.fr
-rw-rw-r-- 1 1000 1000 2.1M Feb  2  2016 train.de
-rw-rw-r-- 1 1000 1000 1.8M Feb  2  2016 train.en
-rw-r--r-- 1 root root 1.2M Jul 15 14:26 training.tar.gz
-rw-rw-r-- 1 1000 1000  75K Feb  2  2016 val.de
-rw-rw-r-- 1 1000 1000  62K Feb  2  2016 val.en
-rw-r--r-- 1 root root  46K Jul 15 14:26 validation.tar.gz


Instruction: "ValueError: Incompatible Language version 13. Must not be between 9 and 12" with Google Colab

Input:
I am trying to build a deep learning model with transformer model architecture. In that case when I am trying to cleaning the dataset following error occurred.
I am using Pytorch and google colab for that case & trying to clean Java methods and comment dataset.
Tested Code
    import re

    from fast_trees.core import FastParser

    parser = FastParser('java')

    def get_cmt_params(cmt: str) -> List[str]:
        '''
        Grabs the parameter identifier names from a JavaDoc comment

        :param cmt: the comment to extract the parameter identifier names from
        :returns: an array of the parameter identifier names found in the given comment
        '''
        params = re.findall('@param+\s+\w+', cmt)
        param_names = []
        for param in params:
            param_names.append(param.split()[1])

        return param_name

Occured Error
Downloading repo https://github.com/tree-sitter/tree-sitter-java to /usr/local/lib/python3.7/dist-packages/fast_trees/tree-sitter-java.
    ---------------------------------------------------------------------------
    ValueError                                Traceback (most recent call last)
    <ipython-input-31-64f6fa6ed39b> in <module>()
          3 from fast_trees.core import FastParser
          4
    ----> 5 parser.set_language = FastParser('java')
          6
          7 def get_cmt_params(cmt: str) -> List[str]:

    3 frames
    /usr/local/lib/python3.7/dist-packages/fast_trees/core.py in FastParser(lang)
         96     }
         97
    ---> 98     return PARSERS[lang]()

    /usr/local/lib/python3.7/dist-packages/fast_trees/core.py in __init__(self)
         46
         47     def __init__(self):
    ---> 48         super().__init__()
         49
         50     def get_method_parameters(self, mthd: str) -> List[str]:

    /usr/local/lib/python3.7/dist-packages/fast_trees/core.py in __init__(self)
         15 class BaseParser:
         16     def __init__(self):
    ---> 17         self.build_parser()
         18
         19     def build_parser(self):

    /usr/local/lib/python3.7/dist-packages/fast_trees/core.py in build_parser(self)
         35         self.language = Language(build_dir, self.LANG)
         36         self.parser = Parser()
    ---> 37         self.parser.set_language(self.language)
         38
         39 # Cell

    ValueError: Incompatible Language version 13. Must not be between 9 and 12

an anybody help me to solve this issue?

Output:
The fast-trees library uses the tree-sitter library and since they recommended using the 0.2.0 version of tree-sitter in order to use fast-trees. Although downgrade the tree-sitter to the 0.2.0 version will not be resolved your problem. I also tried out it by downgrading it.
So, without investing time to figure out the bug in tree-sitter it is better to move to another stable library that satisfies your requirements. So, as your requirement, you need to extract features from a given java code. So, you can use javalang library to extract features from a given java code.

javalang is a pure Python library for working with Java source code.
javalang provides a lexer and parser targeting Java 8. The
implementation is based on the Java language spec available at
http://docs.oracle.com/javase/specs/jls/se8/html/.

you can refer it from - https://pypi.org/project/javalang/0.13.0/
Since javalang is a pure library it will help go forward on your research without any bugs


Instruction: RuntimeError: mat1 and mat2 shapes cannot be multiplied (5376x28 and 784x512)

Input:
Basic Network
class Baseline(nn.Module):
    def __init__(self):
        super().__init__()
        # 5 Hidden Layer Network
        self.fc1 = nn.Linear(28 * 28, 512)
        self.fc2 = nn.Linear(512, 256)
        self.fc3 = nn.Linear(256, 128)
        self.fc4 = nn.Linear(128, 64)
        self.fc5 = nn.Linear(64, 3)

        # Dropout module with 0.2 probbability
        self.dropout = nn.Dropout(p=0.2)
        # Add softmax on output layer
        self.log_softmax = F.log_softmax

    def forward(self, x):
        x = self.dropout(F.relu(self.fc1(x)))
        x = self.dropout(F.relu(self.fc2(x)))
        x = self.dropout(F.relu(self.fc3(x)))
        x = self.dropout(F.relu(self.fc4(x)))

        x = self.log_softmax(self.fc5(x), dim=1)

        return x

Error:
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-48-0030d9c3852c> in <module>
     18         optimizer.zero_grad()
     19         # Make predictions
---> 20         log_ps = model(images)
     21         loss = criterion(log_ps, labels)
     22         #backprop

~/anaconda3/envs/torch/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

<ipython-input-46-09dd06cd0a72> in forward(self, x)
     15
     16     def forward(self, x):
---> 17         x = self.dropout(F.relu(self.fc1(x)))
     18         x = self.dropout(F.relu(self.fc2(x)))
     19         x = self.dropout(F.relu(self.fc3(x)))

~/anaconda3/envs/torch/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

~/anaconda3/envs/torch/lib/python3.8/site-packages/torch/nn/modules/linear.py in forward(self, input)
     91
     92     def forward(self, input: Tensor) -> Tensor:
---> 93         return F.linear(input, self.weight, self.bias)
     94
     95     def extra_repr(self) -> str:

~/anaconda3/envs/torch/lib/python3.8/site-packages/torch/nn/functional.py in linear(input, weight, bias)
   1690         ret = torch.addmm(bias, input, weight.t())
   1691     else:
-> 1692         output = input.matmul(weight.t())
   1693         if bias is not None:
   1694             output += bias

RuntimeError: mat1 and mat2 shapes cannot be multiplied (5376x28 and 784x512)

I tried changing fc1 to self.fc1 = nn.Linear(5376, 512) but I still get RuntimeError: mat1 and mat2 shapes cannot be multiplied (5376x28 and 5376x512).
I then adjusted the whole architecture as follows:
class Baseline(nn.Module):
    def __init__(self):
        super().__init__()
        # 5 Hidden Layer Network
        self.fc1 = nn.Linear(5376, 28)
        self.fc2 = nn.Linear(28, 256)
        self.fc3 = nn.Linear(256, 128)
        self.fc4 = nn.Linear(128, 64)
        self.fc5 = nn.Linear(64, 3)

        # Dropout module with 0.2 probbability
        self.dropout = nn.Dropout(p=0.2)
        # Add softmax on output layer
        self.log_softmax = F.log_softmax

    def forward(self, x):
        x = self.dropout(F.relu(self.fc1(x)))
        x = self.dropout(F.relu(self.fc2(x)))
        x = self.dropout(F.relu(self.fc3(x)))
        x = self.dropout(F.relu(self.fc4(x)))

        x = self.log_softmax(self.fc5(x), dim=1)

        return x

and I still get the following error:
RuntimeError                              Traceback (most recent call last)
<ipython-input-54-0030d9c3852c> in <module>
     18         optimizer.zero_grad()
     19         # Make predictions
---> 20         log_ps = model(images)
     21         loss = criterion(log_ps, labels)
     22         #backprop

~/anaconda3/envs/torch/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

<ipython-input-52-f98f89e15885> in forward(self, x)
     15
     16     def forward(self, x):
---> 17         x = self.dropout(F.relu(self.fc1(x)))
     18         x = self.dropout(F.relu(self.fc2(x)))
     19         x = self.dropout(F.relu(self.fc3(x)))

~/anaconda3/envs/torch/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

~/anaconda3/envs/torch/lib/python3.8/site-packages/torch/nn/modules/linear.py in forward(self, input)
     91
     92     def forward(self, input: Tensor) -> Tensor:
---> 93         return F.linear(input, self.weight, self.bias)
     94
     95     def extra_repr(self) -> str:

~/anaconda3/envs/torch/lib/python3.8/site-packages/torch/nn/functional.py in linear(input, weight, bias)
   1690         ret = torch.addmm(bias, input, weight.t())
   1691     else:
-> 1692         output = input.matmul(weight.t())
   1693         if bias is not None:
   1694             output += bias

RuntimeError: mat1 and mat2 shapes cannot be multiplied (5376x28 and 5376x28)

NB: The input data is of shape torch.Size([64, 3, 28, 28]).
For the purposes of replication:
model = Baseline()
X = torch.rand(64, 3, 28, 28)
model(X)

Output:
I see one issue in the code.
Linear layers do not accept matrices with a 4d shape that you passed into the model.
In order to pass data with torch.Size([64, 3, 28, 28]) through a nn.Linear() layers like you have in your model. You need to flatten the tensor in your forward function like:
# New code
x = x.view(x.size(0), -1)
#Your code
x = self.dropout(F.relu(self.fc1(x)))

...

This will probably help solve the weight matrix error you are getting.
Sarthak Jain


Instruction: Convert keras model to pytorch

Input:
Is there an easy way to convert a model like this from keras to pytorch?
I have the code in keras as following:
from tensorflow.keras import Sequential
from tensorflow.keras.layers import Dense, Dropout
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.regularizers import l2

state_dim = 10
architecture = (256, 256)  # units per layer
learning_rate = 0.0001  # learning rate
l2_reg = 0.00000001  # L2 regularization
trainable = True
num_actions = 3

layers = []
n = len(architecture) # n = 2

for i, units in enumerate(architecture, 1):
    layers.append(Dense(units=units,
                        input_dim=state_dim if i == 1 else None,
                        activation='relu',
                        kernel_regularizer=l2(l2_reg),
                        name=f'Dense_{i}',
                        trainable=trainable))

layers.append(Dropout(.1))
layers.append(Dense(units=num_actions,
                    trainable=trainable,
                    name='Output'))

model = Sequential(layers)
model.compile(loss='mean_squared_error',
            optimizer=Adam(lr=learning_rate))

Which outputs as follow:
Model: "sequential_2"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
Dense_1 (Dense)              (None, 256)               2816
_________________________________________________________________
Dense_2 (Dense)              (None, 256)               65792
_________________________________________________________________
dropout_3 (Dropout)          (None, 256)               0
_________________________________________________________________
Output (Dense)               (None, 3)                 771
=================================================================
Total params: 69,379
Trainable params: 69,379
Non-trainable params: 0
_________________________________________________________________
None

I must admit, I'm a little out of my depth so any advice is appreciated. I'm trying to read through the pytorch docs and will update my question with a possible answer if I manage.

Output:
Here is my best attempt:
state_dim = 10
architecture = (256, 256)  # units per layer
learning_rate = 0.0001  # learning rate
l2_reg = 0.00000001  # L2 regularization
trainable = True
num_actions = 3

import torch
from torch import nn

class CustomModel(nn.Module):
    def __init__(self):

        super().__init__()

        self.layers = nn.Sequential(
            nn.Linear(state_dim, architecture[0]),
            nn.ReLU(),
            nn.Linear(architecture[0], architecture[1]),
            nn.ReLU(),
            nn.Dropout(0.25),
            nn.Linear(architecture[1], num_actions),
        )

    def forward(self, x):
        return self.layers(x)

model = CustomModel()
criterion = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

It outputs a promising looking output:
CustomModel(
    (layers): Sequential(
        (0): Linear(in_features=10, out_features=256, bias=True)
        (1): ReLU()
        (2): Linear(in_features=256, out_features=256, bias=True)
        (3): ReLU()
        (4): Dropout(p=0.25, inplace=False)
        (5): Linear(in_features=256, out_features=3, bias=True)
    )
)

However a few items are still left unanswered:

are the activations in the right place?
how do we add a kernel_regularizer = l2(l2_reg) to the first two Linear/Dense layers?
and how do we make the layers trainable?

Any input appreciated.


Instruction: How to make a custom torchvision transform?

Input:
I have a function that changes image pixels with 20% chance, but not sure how to make it work in transforms.Compose([]). Please help!
def random_t(img):
    im = Image.open(img)
    pixelMap = im.load()
    pixelMap_list = []

    for i in range(im.size[0]):
        for j in range(im.size[1]):
            randNum = random.uniform(0, 1)
            if randNum < 0.2:  # 20% chance of pixel change
                pixelMap[i, j] = (random.randint(0, 255), random.randint(0, 255), random.randint(0, 255))
            pixelMap_list.append(pixelMap[i, j])
        else:
            pixelMap[i, j] = pixelMap[i, j]

    return im

I think it should have a format like this.. this is from pytorch library.
class custom_augmentation(object):
    def __init__(self, p):
        self.p = p # it should be the probability of random pixel

    def __call__(self, img):
        return None # Not sure how to make random_t in here

    def __repr__(self):
        return "custom augmentation"

fixed code:
class custom_augmentation(object):
    def __init__(self, p=0.5):
        self.p = p

    def __call__(self, img):
        pixelMap = img.load()
        for i in range(img.size[0]):
            for j in range(img.size[1]):
                if torch.rand(1) < self.p:
                    pixelMap[i, j] = (random.randint(0, 255), random.randint(0, 255), random.randint(0, 255))
                else:
                    pixelMap[i, j] = pixelMap[i, j]

        return img # Not sure how to make random_t in here

    def __repr__(self):
        return "custom augmentation"

Output:
You need to do your operations on img and then return it. For a good example of how to create custom transforms just check out how the normal torchvision transforms are created like over here:
This is the github where torchvision.transforms like transforms.Resize(), transforms.ToTensor(), transforms.RandomHorizontalFlip() have their code. Look at these transforms like RandomHorizontalFlip() to see how to introduce a probability that a transform will happen etc.
https://github.com/pytorch/vision/blob/master/torchvision/transforms/transforms.py
Sarthak Jain


Instruction: Imorting zero_gradients from torch.autograd.gradcheck

Input:
I want to replicate the code here, and I get the following error while running in Google Colab?

ImportError: cannot import name 'zero_gradients' from
'torch.autograd.gradcheck'
(/usr/local/lib/python3.7/dist-packages/torch/autograd/gradcheck.py)

Can someone help me with how to solve this?

Output:
This seems like it's using a very old version of PyTorch, the function itself is not available anymore. However, if you look at this commit, you will see the implementation of zero_gradients. What it does is simply zero out the gradient of the input:
def zero_gradients(i):
    for t in iter_gradients(i):
        t.zero_()

Then zero_gradients(x) should be the same as x.zero_grad(), which is the current API, assuming x is a nn.Module!
Or it's just:
if x.grad is not None:
    x.grad.zero_()


Instruction: PyTorch: Image not displaying properly

Input:
I have the following code portion:
dataset = trainDataset()
train_loader = DataLoader(dataset,batch_size=1,shuffle=True)

device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

images = []
image_labels = []

for i, data in enumerate(train_loader,0):
    inputs, labels = data
    inputs, labels = inputs.to(device), labels.to(device)
    inputs, labels = inputs.float(), labels.float()
    images.append(inputs)
    image_labels.append(labels)

image = images[7]
image = image.numpy()
image = image.reshape(416,416,3)
img = Image.fromarray(image,'RGB')
img.show()

The issue is that the image doesn't display properly. For instance, the dataset I have contains images of cats and dogs. But, the image displayed looks as shown below. Why is that?

EDIT 1
So, after @flawr's nice explanation, I have the following:
image = images[7]
image = image[0,...].permute([1,2,0])
image = image.numpy()
img = Image.fromarray(image,'RGB')
img.show()

And, the image looks as shown below. Not sure if it is a Numpy thing or the way the image is represented and displayed? I would like to also kindly note that I get a different display of the image at every run, but it is pretty much something close to the image displayed below.

EDIT 2
I think the issue now is with how to represent the image. By referring to this solution, I now get the following:
image = images[7]
image = image[0,...].permute([1,2,0])
image = image.numpy()
image = (image * 255).astype(np.uint8)
img = Image.fromarray(image,'RGB')
img.show()

Which produces the following image as expected :-)

Output:
In pytorch you usually represent pictures with tensors of shape
(channels, height, width)

You then seem to reshape it to what you expect would be
(height, width, channels)

Note that these tensors or arrays are actually stored as 1d "array", and the multiple dimensions just come from defining strides (check out How to understand numpy strides for layman?).
In your particular case this means that consecutive values (that were basically values of the same color channela and the same row) are now interpreted as different colour channels.
So let's say you have a 2x2 image with 3 color channels. Let's say it is a chessboard pattern. In pytorch that would looks something like the following array of shape (3, 2, 2):
[[[1,0],[0,1]],[[1,0],[0,1]],[[1,0],[0,1]]]

The underlaying internal array is just
[  1,0 , 0,1  ,  1,0 , 0,1  ,  1,0 , 0,1  ]

So reshaping to (2, 2, 3) would look like so:
[[[1,0,0],[1,1,0]],[[0,1,1],[0,0,1]]]

which immediately shows how the image will be completely jumbled. Reshaping really just means setting the brackets in different places!
So what you probably want instead of reshape is permute([1, 2, 0]), (or in numpy called transpose) which will actually rearrange the data.


Instruction: How does pytorch perform the reverse-differentiation given an indexed version of a tensor in the feedforward step?

Input:
Some of this code was adapted from the book Deep learning with Pytorch
Script: Linear regression (trying to predict t_c given t_u)
t_c = torch.tensor([0.5, 14.0, 15.0, 28.0, 11.0, 8.0,
                    3.0, -4.0, 6.0, 13.0, 21.0])
t_u = torch.tensor([35.7, 55.9, 58.2, 81.9, 56.3, 48.9,
                    33.9, 21.8, 48.4, 60.4, 68.4])

def model(t_u, w, b):
    return w * t_u + b

def loss_fn(t_p, t_c):
    squared_diffs = (t_p - t_c)**2
    return squared_diffs.mean()

params = torch.tensor([1.0, 0.0], requires_grad=True)

loss = loss_fn(model(t_u, params[0], params[1]), t_c)
loss.backward()

print(params.grad)

Here I am passing in the 0th and 1st index of params as an input to the model function, which performs scalar-to-vector multiplication and addition.
My question is, what is PyTorch exactly doing to compute the gradients of the params tensor? The "feedforward" step uses two subtensors of the params tensor, rather than separate tensors for bias and weight, which is what I am familiar with.
My guess is: params[0] and params[1] are both references to elements in params, and they both have their own distinct gradients stored somewhere in the params.grad. So the .backward() call is treating params[0] and params[1] as new individual tensors (as if we temporarily had two separate tensors -weight and bias) and updates their gradients (params[0].grad, params[1].grad), hence updating the params.grad since they are references to it.

Output:
The main idea here is that the indexing operation returns a new view of the tensor. If you are not using in-place operations (+=, -=, etc.), the "view" thing does not really matter and you can consider it as just another tensor.
In that case, the indexing operation is no different from other operations like addition or matrix-multiplication -- input (original tensor), output (selected tensor), and gradient (1 if selected, zero otherwise*). Then back-propagation happens as usual.
* More specifically, the gradient of an input entry with respect to an output entry is 1 if the output entry is selected from the input entry; 0 otherwise.
EDIT:
Maybe it's easier to see it this way:
a = d_params[0]
c = W*a+b
--------------------------
dc/d_params
= dc/{d_params[0], d_params[1], d_params[2], ...}
--------------------------
dc/d_params[0]
= dc/da * da/d_params[0]
= dc/da * 1
= W
--------------------------
dc/d_params[1], dc/d_params[2], ... = 0


Instruction: How to disable tqdm's progressbar and keep only the text info in Pytorch Lightning (or in tqdm in general)

Input:
I am working on Pytorchlightning and tqdm's progressbar is very buggy, it keep resizing back and forth from short to long, making reading the logging text so unpleasant, I realized that the progressbar is not really necessary and would like to keep only the info about the current epoch, current batch, accuracy, loss, etc.
From my searching it seems like you can disable whole tqdm display(progressbar and text), but how can I selectively disable only progressbar but not the text?

Output:
The tqdm way to disable the "meter" (while retaining display of stats) is to set ncols=0 and dynamic_ncols=False (see tqdm documentation).
The way to customize the default progress bar behavior in pytorch_lightning is to pass a custom ProgressBar in as a callback when building the Trainer.
Putting the two together, if you wanted to modify the progress bar during training you could do something like the following:
import pytorch_lightning as pl
from pytorch_lightning.callbacks import ProgressBar

class MeterlessProgressBar(ProgressBar):

    def init_train_tqdm(self):
        bar = super().init_train_tqdm()
        bar.dynamic_ncols = False
        bar.ncols = 0
        return bar

bar = MeterlessProgressBar()
trainer = pl.Trainer(callbacks=[bar])

You can separately customize for the sanity check, prediction, validation, and test by overriding: init_sanity_tqdm, init_predict_tqdm, init_validation_tqdm, and init_test_tqdm respectively. (If you want a quick and dirty way to do something to all progress bars, you could consider overriding the _update_bar method instead.)


Instruction: Pytorch Custom Loss Function with If Statement

Input:
I am trying to create a custom loss function in Pytorch that evaluates each element of a tensor with an if statement and acts accordingly.
def my_loss(outputs,targets,fin_val):
    if (outputs.detach()-fin_val.detach())*(targets.detach()-fin_val.detach())<0:
        loss=3*(outputs-targets)**2
    else:
        loss=0.3*(outputs-targets)**2
    return loss

I have also tried:
def my_loss(outputs,targets,fin_val):
    if torch.gt((outputs.detach()-fin_val.detach())*(targets.detach()-fin_val.detach()),0):
        loss=0.3*(outputs-targets)**2
    else:
        loss=3*(outputs-targets)**2
    return loss

In both cases, I get the following error:
RuntimeError: Boolean value of Tensor with more than one value is ambiguous

TIA

Output:
You are getting this error because the condition you are passing to the if statement is not a boolean but a tensor of booleans. Just check what's the nature of (outputs.detach()-fin_val.detach())*(targets.detach()-fin_val.detach())<0, it is a tensor!
What you should be looking to do instead is handling this in vectorized form. You can use torch.where which is built for this use:
torch.where(condition=(outputs - fin_val)*(targets - fin_val) < 0,
            x=3*(outputs-targets)**2,
            y=0.3*(outputs-targets)**2)

This will return a tensor of "xs" and "ys" based on the point-wise condition tensor condition. Then, you could average it depending on your needs to get an actual loss value.


Instruction: How to print the adjusting learning rate in Pytorch?

Input:
While I use torch.optim.Adam and exponential decay_lr in my PPO algorithm:
        self.optimizer = torch.optim.Adam([
            {'params': self.policy.actor.parameters(), 'lr': lr_actor},
            {'params': self.policy.critic.parameters(), 'lr': lr_critic}
        ])
        self.scheduler = torch.optim.lr_scheduler.ExponentialLR(self.optimizer, self.GAMMA)

The initial lr=0.1, and GAMMA=0.9.
Then I print the lr in my epoch dynamiclly with:
            if time_step % update_timestep == 0:
                ppo_agent.update()
                print(f'__________start update_______________')
                print(ppo_agent.optimizer.state_dict()['param_groups'][0]['lr'])

But, something wrong with this, and the bug is:
  File "D:\Anaconda\lib\site-packages\torch\distributions\beta.py", line 36, in __init__
    self._dirichlet = Dirichlet(concentration1_concentration0, validate_args=validate_args)
  File "D:\Anaconda\lib\site-packages\torch\distributions\dirichlet.py", line 52, in __init__
    super(Dirichlet, self).__init__(batch_shape, event_shape, validate_args=validate_args)
  File "D:\Anaconda\lib\site-packages\torch\distributions\distribution.py", line 53, in __init__
    raise ValueError("The parameter {} has invalid values".format(param))
ValueError: The parameter concentration has invalid values

Then, if I delete the "print()" sentence, it does work well!
So, it's bothering me very much.

Output:
You can get the learning rate like this:
self.optimizer.param_groups[0]["lr"]


Instruction: Problem with variable when working with python

Input:
Hello everyone im trying to work with the Digit tactile sensor with the PyTouch library so when i try to run the contact area code example i get this error
DigitSensor with SensorDataSources.RAW data source
Traceback (most recent call last):
File "contactarea.py", line 31, in <module>
extract_surface_contact()
File "contactarea.py", line 17, in extract_surface_contact
major, minor = pt.ContactArea(sample_img, base=base_img)
File "/home/ayoub.hichri/.local/lib/python3.8/site-
packages/pytouch/tasks/contact_area.py", line 32, in __call__
) = self._compute_contact_area(contours, self.contour_threshold)
File "/home/ayoub.hichri/.local/lib/python3.8/site-
packages/pytouch/tasks/contact_area.py", line 107, in _compute_contact_area
return poly, major_axis, major_axis_end, minor_axis, minor_axis_end
UnboundLocalError: local variable 'poly' referenced before assignment

this is the code in file contact_area.py
    def _compute_contact_area(self, contours, contour_threshold):
    for contour in contours:
        if len(contour) > contour_threshold:
            ellipse = cv2.fitEllipse(contour)
            poly = cv2.ellipse2Poly(
                (int(ellipse[0][0]), int(ellipse[0][1])),
                (int(ellipse[1][0] / 2), int(ellipse[1][1] / 2)),
                int(ellipse[2]),
                0,
                360,
                5,
            )
            center = np.array([ellipse[0][0], ellipse[0][1]])
            a, b = (ellipse[1][0] / 2), (ellipse[1][1] / 2)
            theta = (ellipse[2] / 180.0) * np.pi
            major_axis = np.array(
                [center[0] - b * np.sin(theta), center[1] + b * np.cos(theta)]
            )
            minor_axis = np.array(
                [center[0] + a * np.cos(theta), center[1] + a * np.sin(theta)]
            )
            major_axis_end = 2 * center - major_axis
            minor_axis_end = 2 * center - minor_axis
    return poly, major_axis, major_axis_end, minor_axis, minor_axis_end

and this is the python code for my main file which im trying to run
     import pytouch
     from pytouch.handlers import ImageHandler
     from pytouch.sensors import DigitSensor
     from pytouch.tasks import ContactArea

def extract_surface_contact():
base_img_path = "/home/../Documents/digit.png"
sample_img_path = "/home/../Documents/Digit2.png"

base_img = ImageHandler(base_img_path).nparray
sample_img = ImageHandler(sample_img_path).nparray
sample_img_2 = sample_img.copy()

# initialize with default configuration of ContactArea task
pt = pytouch.PyTouch(DigitSensor, tasks=[ContactArea])
major, minor = pt.ContactArea(sample_img, base=base_img)

print("Major Axis: {0}, minor axis: {1}".format(*major, *minor))
ImageHandler.save("surface_contact_1.png", sample_img)

# initialize with custom configuration of ContactArea task
contact_area = ContactArea(base=base_img, contour_threshold=10)
major, minor = contact_area(sample_img_2)

print("Major Axis: {0}, minor axis: {1}".format(*major, *minor))
ImageHandler.save("surface_contact_2.png", sample_img_2)

if __name__ == "__main__":
extract_surface_contact()

Thanks in advance for anyone who helps me

Output:
Seems like the condition len(contour) > contour_threshold inside _compute_contact_area is never matched so the variable poly is never defined.
I recommend trying to print the length of contour before the if statement to check the values. If you want it to work even if the condition isn't matched just declare the variable at the start of your function with an empty value (poly = None)


Instruction: Key already registered with the same priority: GroupSpatialSoftmax

Input:
I get this error:
"Key already registered with the same priority: GroupSpatialSoftmax"
when i run:
    import torch

Though I've installed the pytorch package through the pycharm settings > python interpreter.
Does anyone know how can I solve it?
Thanks

Output:
Solved it myself!
I uninstalled the pytorch package and re-installed it so now it works


Instruction: Defining Metrics on SageMaker to CloudWatch

Input:
From AWS Sagemaker Documentation, In order to track metrics in cloudwatch for custom ml algorithms (non-builtin), I read that I have to define my estimaotr as below.
But I am not sure how to alter my training script so that the metric definitions declared inside my estimators can pick up these values.
estimator =
                Estimator(image_name=ImageName,
                role='SageMakerRole',
                instance_count=1,
                instance_type='ml.c4.xlarge',
                k=10,
                sagemaker_session=sagemaker_session,
                metric_definitions=[
                   {'Name': 'train:error', 'Regex': 'Train_error=(.*?);'},
                   {'Name': 'validation:error', 'Regex': 'Valid_error=(.*?);'}
                ]
            )

In my training code, I have
    for epoch in range(1, args.epochs + 1):
        total_loss = 0
        model.train()
        for step, batch in enumerate(train_loader):
            b_input_ids = batch[0].to(device)
            b_input_mask = batch[1].to(device)
            b_labels = batch[2].to(device)
            model.zero_grad()

            outputs = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask, labels=b_labels)
            loss = outputs[0]

            total_loss += loss.item()
            loss.backward() # Computes the gradients
            torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) # Clip for error prevention
            # modified based on their gradients, the learning rate, etc.
            optimizer.step() # Back Prop
logger.info("Average training loss: %f\n", total_loss / len(train_loader))

Here, I want the train:error to pick up total_loss / len(train_loader) but I am not sure how to assign this.

Output:
You have to define a regex to capture that pattern, try with this:
{'Name': 'Average training loss', 'Regex': 'Average training loss = ([0-9\.]+)'}

You can try the regex in tool like this and see what happens.