Instruction: Running PyTorch multiprocessing in a Docker container with Gunicorn worker manager

Input:
I am trying to deploy a service on GCP. It's a Docker container that uses Gunicorn for worker management.
The code runs a torch.multiprocessing.process to run a POST response as a background process.

This works if I run the script using a python3 command. But hangs when using Gunicorn.

My understanding is that CUDA needs threadsafe multiprocessing and that is why torch has its own implementation. When we set up Gunicorn to manage workers, this may be causing some conflict or thread safety issues.

Has anyone come across this before. Would there be a different worker manager that I could be using?

In Dockerfile: CMD gunicorn -w 1 -t 6000 -b 0.0.0.0:8080 --timeout 6000 --preload app_script:app - this is how i am running the app in docker. So yes I am using preload. And the issue happens even if I run the docker container locally so its not just a gcp situation

p=torch.multiprocessing.Process(target=my_function args=(args, )) . p.start() - this is how a post call is getting handled.

Output:
I spent a lot of time investigating a similar issue.
Pytorch calls were stuck when running on a docker container with gunicorn.

The solution that worked for me was removing the --preload flag from the Docker gunicorn command.


Instruction: Stacking tensors in a list of tuples of tensors

Input:
I have a list of tuples of PyTorch tensors. It looks like this:

[
    (tensor([1, 2, 3]),  tensor([4, 5, 6, 7]),  tensor([8])),
    (tensor([9, 10,11]), tensor([11,12,13,14]), tensor([15])),
    (tensor([16,17,18]), tensor([19,20,21,22]), tensor([23])),
    ...
]

Tensors in each column (that is, tensors that position k of their respective tuple) share the same shape. I want to stack the tensors in each column so that I end up with a single tuple, each value being the tensors concatenated along the dimension of the column.

In this case, the output tuple would have three values, and look like this:

(
 tensor([[1,2,3], [9,10,11], [16,17,18]]),

 tensor([[4,5,6,7], [11,12,13,14], [19,20,21,22]],

 tensor([[8],[15],[23])
)

This is a made-up example. I want to do this for tuples of any length, and tensors of arbitrary size. What is the best way to do this type of concatenation quickly using PyTorch?

Output:
If anyone gets themselves into the same convoluted scenario, I was able to solve it with a lovely one-liner:
tuple(map(torch.stack, zip(*x)))

In this case, x is the original list I mentioned above. This line of code transforms x into the exact desired format.


Instruction: How can I save model weights to mlflow tracking sever using pytorch-lightning?

Input:
I would like to save model weights to mlflow tracking using pytorch-lightning.
pytorch-lightning supports logging.
However, it seems that saving model weights as a artifact on mlflow is not supported.

At first, I planed to override ModelCheckpoint class to do it, but I found it is difficult for me because of complex Mixin operations.

Anybody knows simple way to accomplish it?

Output:
As @xela said, you can use the experiment object of the mlflow logger to log artifacts.
In case you want to frequently log model weights during training, you could extend ModelCheckpoint:

class MLFlowModelCheckpoint(ModelCheckpoint):
    def __init__(self, mlflow_logger, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.mlflow_logger = mlflow_logger

    @rank_zero_only
    def on_validation_end(self, trainer, pl_module):
        super().on_validation_end(trainer, pl_module)
        run_id = self.mlflow_logger.run_id
        self.mlflow_logger.experiment.log_artifact(run_id, self.best_model_path)

And then use in your training code
mlflow_logger = MLFlowLogger()
checkpoint_callback = MLFlowModelCheckpoint(mlflow_logger)
trainer = pl.Trainer(checkpoint_callback=checkpoint_callback, logger=mlflow_logger)


Instruction: Cannot install pytorch in a virtualenv on windows

Input:
I know there are a few topics about that on this website but still, I can't find the solution.

So here is what I did :
I created a projecton Visual Studio 19 for python.
I added an virtual environment with Python 3.7 using the file requirements.txt

It contains

mypy==0.750
pylint==2.4.4
pytest==5.3.1
matplotlib==3.1.1
torch==1.3.1
tensorflow==2.1.0rc0

This requirement file works perfectly well on linux. But on vs19 I get the following :

ERROR: Could not find a version that satisfies the requirement torch==1.3.1 (from -r E:\Documents\Blub\Granolar\requirements.txt (line 5)) (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)`

Since I don't want to downgrade to 0.1.2 (obviously) how can I fix it?

Output:
I went to the PyTorch documentation on how to "Start Locally" and selected what seems to be your environment:

PyTorch Build: Stable (1.3)
Your OS: Windows
Package: Pip
Language: Python 3.7
CUDA: None

The resulting instruction I got back as a result is:

pip3 install torch==1.3.1+cpu torchvision==0.4.2+cpu -f https://download.pytorch.org/whl/torch_stable.html

So most likely you could modify your requirements.txt like this:

mypy==0.750
pylint==2.4.4
pytest==5.3.1
matplotlib==3.1.1
torch==1.3.1+cpu --find-links https://download.pytorch.org/whl/torch_stable.html
tensorflow==2.1.0rc0


Instruction: why is my Neural Network stuck at high loss value after the first epochs

Input:
I'm doing regression using Neural Networks. It should be a simple task for NN to do, I have 10 features and 1 output that I want to predict.I’m using pytorch for my project but my Model is not learning well. the loss start with a very high value (40000), then after the first 5-10 epochs the loss decrease rapidly to 6000-7000 and then it stuck there, no matter what I make. I tried even to change to skorch instead of pytorch so that I can use cross validation functionality but that also didn’t help. I tried different optimizers and added layers and neurons to the network but that didn’t help, it stuck at 6000 which is a very high loss value. I’m doing regression here, I have 10 features and I’m trying to predict one continuous value. that should be easy to do that’s why it is confusing me more.

here is my network:
I tried here all the possibilities from making more complex architectures like adding layers and units to batch normalization, changing activations etc.. nothing have worked

class BearingNetwork(nn.Module):
    def __init__(self, n_features=X.shape[1], n_out=1):
        super().__init__()
        self.model = nn.Sequential(

            nn.Linear(n_features, 512),
            nn.BatchNorm1d(512),
            nn.LeakyReLU(),
            nn.Linear(512, 64),
            nn.BatchNorm1d(64),
            nn.LeakyReLU(),
            nn.Linear(64, n_out),
#             nn.LeakyReLU(),
#             nn.Linear(256, 128),
#             nn.LeakyReLU(),
#             nn.Linear(128, 64),
#             nn.LeakyReLU(),
#             nn.Linear(64, n_out)
        )

    def forward(self, x):
        out = self.model(x)
        return out

and here are my settings:
using skorch is easier than pytorch. here I'm monitoring also the R2 metric and I made RMSE as a custom metric to also monitor the performance of my model. I also tried the amsgrad for Adam but that didn't help.

R2 = EpochScoring(r2_score, lower_is_better=False, name='R2')
explained_var_score = EpochScoring(EVS, lower_is_better=False, name='EVS Metric')
custom_score = make_scorer(RMSE)
rmse = EpochScoring(custom_score, lower_is_better=True, name='rmse')

bearing_nn = NeuralNetRegressor(

    BearingNetwork,
    criterion=nn.MSELoss,
    optimizer=optim.Adam,
    optimizer__amsgrad=True,
    max_epochs=5000,
    batch_size=128,
    lr=0.001,
    train_split=skorch.dataset.CVSplit(10),
    callbacks=[R2, explained_var_score, rmse, Checkpoint(), EarlyStopping(patience=100)],
    device=device

)

I also standardize the Input values.

my Input have the shape:

torch.Size([39006, 10])

and shape of output is:

torch.Size([39006, 1])

I’m using 128 as my Batch_size but I also tried other values like 32, 64, 512 and even 1024. Although normalizing output is not necessary but I also tried that and It didn’t work when I predict values, the loss is high. Please someone help me on this, I would appreciate every helpful advice. I ll also add a screenshot of my training and val losses and metrics over epochs to visualize how the loss is decreasing in the first 5 epochs and then it stays like forever at the value 6000 which is a very high value for a loss.

Output:
considering that your training and dev loss are decreasing over time, it seems like your model is training correctly. With respect to your worry regarding your training and dev loss values, this is entirely dependent on the scale of your target values (how big are your target values?) and the metric used to compute the training and dev losses. If your target values are big and you want smaller train and dev loss values, you can normalise the target values.

From what I gather with respect to your experiments as well as your R2 scores, it seems that you are looking for a solution in the wrong area. To me, it seems like your features aren't strong enough considering that your R2 scores are low, which could mean that you have a data quality issue. This would also explain why your architecture tuning has not improved your model's performance as it is not your model that is the issue. So if I were you, I would think about what new useful features I could add and see if that helps. In machine learning, the general rule is that models are only as good as the data that they are trained on. I hope this helps!


Instruction: How to install PyTorch on Python 3.7 / Windows 10 with pip

Input:
I am trying to install PyTorch 1.3 using pip :

pip3 install torch==1.3.1+cpu torchvision==0.4.2+cpu -f https://download.pytorch.org/whl/torch_stable.html

I got the command line from this page :
https://pytorch.org/get-started/locally/#start-locally

It fails, with the following error message :

ERROR: Could not find a version that satisfies the requirement
torch==1.3.1+cpu (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
ERROR: No matching distribution found for torch==1.3.1+cpu

Using verbose mode (-v) give a little more info :

Skipping link: unsupported archive format: .html:
   https://download.pytorch.org/whl/torch_stable.html (from -f)
Analyzing links from page https://pypi.org/simple/torch/

It seems pip cannot use the url I provide and will get packages from another URL.
That one support at best Python 3.6. Since I have installed 3.7.5 (which is support by URL initially provided) none of the packages match and it fails.

I could downgrade to 3.6 but since 3.7 is officially supported I would like to try to fix that issue first.

EDIT : pip is up to date. I got exact same error on both Windows 10 and 7 64 bit machines.
Also: the PyTorch page says  :

  Please ensure that you have met the prerequisites below (e.g., numpy)

But AFAIK does not list any library in that section. During my tests, numpy was installed and up to date.

As requested, here is the full output of pip :
https://pastebin.com/8Wpqp6du

Output:
You error is being triggered because the version 1.3.1+cpu doesn't exist.

If you go to pytorch.org you will be able to select a version of pytorch and your OS, where it will give you a command to install pyTorch correctly, for Python 3.7 and PIP use the following:

pip3 install --find-links https://download.pytorch.org/whl/torch_stable.html torch==1.3.1 torchvision==0.4.2

Remember to check that the pip3 binary you are using is the one installed over Python 3.7.

Also, mind that the documentation you are pointing at is for MacOS.


Instruction: How I make pytorch read the numpy format?

Input:
I'm trying to create a code with PyTorch and Keras that uses the BERT algorithm to detect fake news, but I got an error tells me:

can't convert np.ndarray of type numpy.bool_.
The only supported types are: double, float, float16, int64, int32, and uint8.

Please access the code on my Google Codelab. The error can be seen in the last cell.

The only requirement for running it is downloading a CSV file for the training process.

Output:
I can't confirm but I believe your problem will be solved by changing:

train_y = np.array(train_labels) == 'fake'
test_y = np.array(test_labels) == 'fake'

to:

train_y = (np.array(train_labels) == 'fake').astype(int)
test_y = (np.array(test_labels) == 'fake').astype(int)

The train_y data is currently an array of type Bool (True or False) and the tensor needs and int (0 or 1).


Instruction: How to make a PyTorch Distribution on GPU

Input:
Is it possible to make the PyTorch distributions create their samples directly on GPU.

If I do

from torch.distributions import Uniform, Normal
normal = Normal(3, 1)
sample = normal.sample()

Then sample will be on CPU. Of course it is possible to do sample = sample.to(torch.device("cuda")) to make it on GPU. But is there a way to have the sample go directly to GPU without first creating it on CPU?

PyTorch distributions inherit from Object, not nn.Module so it does not have a to method the put the distribution instance on GPU.

Any ideas?

Output:
Distributions use the reparametrization trick. Thus giving size 0 tensors which are on GPU to the distribution constructor works. As follows:

normal = Normal(torch.tensor(0).to(device=torch.device("cuda")), torch.tensor(1).to(device=torch.device("cuda")))


Instruction: pytorch net creation does not generate weights if done in a list

Input:
I'm creating a net in pytorch by writing a class called MyNet with init() and forward() method.
If I create a layer in init() like:

self.fc = nn.Linear(5, 10)

everything works fine and

net = MyNet()
paramL = list(net.parameters())

gives me a list with some weights inside.
However, if I create layers in the following way

self.layerL = [nn.Linear(5,10)]

something seems to go wrong, since list(net.parameters()) now gives an empty list :-(

Any idea what I'm doing wrong ??

Many thanks

Output:
In simple terms, this is because it is not a torch.nn object.
For this use torch.nn.Sequential. For example,

self.Layer = torch.nn.Sequential(nn.Linear(5,10), nn.Linear(10,10), ...)


Instruction: Wrong value of standard deviation

Input:
Hello I am trying to evaluate the standard deviation and the mean of the dataset MNIST and I get a wrong value for the standard deviation. Here is my code :

import torch
from torchvision import datasets, transforms
import torch.nn.functional as F

loader = torch.utils.data.DataLoader(datasets.MNIST(
'../data', train=True, download=True, transform=transform1),
                     batch_size=32,
                     num_workers=0,
                     shuffle=False)

mean = 0.
std = 0.
for images, _ in loader:
    batch_samples = images.size(0)
    images = images.view(batch_samples, images.size(1), -1)
    mean += images.mean(2).sum(0)
    std += images.std(2).sum(0)

mean /= len(loader.dataset)
std /= len(loader.dataset)

print("The mean is ", mean)
print("The standard deviation is ", std)

My problem is the following, I get as mean the value 0.1307 and for the standard deviation the value 0.3015 instead of 0.3081. I suppose I have an error in my code but I don't see where.

Could you help me please ?

Thank you very much !

Output:
torch.std uses the batches mean as part of the computation so it's not the same as using torch.std on the entire dataset since that would use a different mean. We can use the following well known expression for variance to get the desired result

Var(X) = E[X**2] - E[X]**2

mean = 0.
mean_square = 0.
samples = 0
for images, _ in loader:
    batch_samples = images.size(0)
    images = images.view(batch_samples, images.size(1), -1)
    mean += images.mean(2).sum(0)
    mean_square += (images**2).mean(2).sum(0)
    samples += images.size(2) * images.size(0)

mean /= len(loader.dataset)
mean_square /= len(loader.dataset)

# extra scale factor for unbias std estimate (it's effectively 1.0)
scale = samples / (samples - 1)
std = torch.sqrt((mean_square - mean**2) * scale)

print("The mean is ", mean)
print("The standard deviation is ", std)

Of course in the special case of the torchvision MNIST dataset you could just directly compute mean and standard deviation...

mean = torch.mean(loader.dataset.data.float() / 255.0)
std = torch.std(loader.dataset.data.float() / 255.0)


Instruction: windows spyder invalid syntax error while running py file

Input:
I am trying to run the last example from the page. I have cloned the repository in the directory C:/Users/nn/Desktop/BERT/transformers-master. I am on windows machine and using spyder IDE. Why i do get below error and how could i resolve it? How do i input the initial part of the poem?

import os

os.chdir('C:/Users/nn/Desktop/BERT/transformers-master/examples')
os.listdir()# It shows run_generation.py file

python run_generation.py \
    --model_type=gpt2 \
    --length=100 \
    --model_name_or_path=gpt2 \

python run_generation.py \
    --model_type=gpt2 \
    --length=100 \
    --model_name_or_path=gpt2 \
  File "<ipython-input-10-501d266b0e64>", line 1
    python run_generation.py \
                        ^
SyntaxError: invalid syntax

I went to command prompt and tried below

cd C:/Users/nn/Desktop/BERT/transformers-master/examples
python3 run_generation.py \--model_type=gpt2 \--length=100 \--model_name_or_path=gpt2 \--promt="Hello world"

nothing happens :(

when i try the same with python command i get an error as below :(

python run_generation.py \--model_type=gpt2 \--length=100 \--model_name_or_path=gpt2 \--promt="Hello world"
2019-12-04 11:23:36.345648: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_100.dll'; dlerror: cudart64_100.dll not found
2019-12-04 11:23:36.352875: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
usage: run_generation.py [-h] --model_type MODEL_TYPE --model_name_or_path
                         MODEL_NAME_OR_PATH [--prompt PROMPT]
                         [--padding_text PADDING_TEXT] [--xlm_lang XLM_LANG]
                         [--length LENGTH] [--num_samples NUM_SAMPLES]
                         [--temperature TEMPERATURE]
                         [--repetition_penalty REPETITION_PENALTY]
                         [--top_k TOP_K] [--top_p TOP_P] [--no_cuda]
                         [--seed SEED] [--stop_token STOP_TOKEN]
run_generation.py: error: the following arguments are required: --model_type, --model_name_or_path

####update 2 ------------------

I followed suggestions in the comments and it worked. It seems that the code downloads 3 files.

Can i copy those files manually so that I dont have to rely on downloading them every time in a temp folder?
Where should i store those files? which folder location? would it be C:\Users\nnn\Desktop\BERT\transformers-master\examples - same as run_generation.py file?

abc

C:\Users\nnn\Desktop\BERT\transformers-master\examples>python run_generation.py --model_type=gpt2 --length=100 --model_name_or_path=gpt2 --prompt="My job is"

2019-12-12 11:11:57.740810: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_100.dll'; dlerror: cudart64_100.dll not found
2019-12-12 11:11:57.748330: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
12/12/2019 11:12:01 - INFO - transformers.file_utils -   https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-vocab.json not found in cache or force_download set to True, downloading to C:\Users\nnn\AppData\Local\Temp\tmpt_29gyqi
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1042301/1042301 [00:00<00:00, 2275416.04B/s]
12/12/2019 11:12:02 - INFO - transformers.file_utils -   copying C:\Users\nnn\AppData\Local\Temp\tmpt_29gyqi to cache at C:\Users\nnn\.cache\torch\transformers\f2808208f9bec2320371a9f5f891c184ae0b674ef866b79c58177067d15732dd.1512018be4ba4e8726e41b9145129dc30651ea4fec86aa61f4b9f40bf94eac71
12/12/2019 11:12:02 - INFO - transformers.file_utils -   creating metadata file for C:\Users\nnn\.cache\torch\transformers\f2808208f9bec2320371a9f5f891c184ae0b674ef866b79c58177067d15732dd.1512018be4ba4e8726e41b9145129dc30651ea4fec86aa61f4b9f40bf94eac71
12/12/2019 11:12:02 - INFO - transformers.file_utils -   removing temp file C:\Users\nnn\AppData\Local\Temp\tmpt_29gyqi
12/12/2019 11:12:03 - INFO - transformers.file_utils -   https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-merges.txt not found in cache or force_download set to True, downloading to C:\Users\nnn\AppData\Local\Temp\tmpj1_y4sn8
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 456318/456318 [00:00<00:00, 1456594.78B/s]
12/12/2019 11:12:03 - INFO - transformers.file_utils -   copying C:\Users\nnn\AppData\Local\Temp\tmpj1_y4sn8 to cache at C:\Users\nnn\.cache\torch\transformers\d629f792e430b3c76a1291bb2766b0a047e36fae0588f9dbc1ae51decdff691b.70bec105b4158ed9a1747fea67a43f5dee97855c64d62b6ec3742f4cfdb5feda
12/12/2019 11:12:03 - INFO - transformers.file_utils -   creating metadata file for C:\Users\nnn\.cache\torch\transformers\d629f792e430b3c76a1291bb2766b0a047e36fae0588f9dbc1ae51decdff691b.70bec105b4158ed9a1747fea67a43f5dee97855c64d62b6ec3742f4cfdb5feda
12/12/2019 11:12:03 - INFO - transformers.file_utils -   removing temp file C:\Users\nnn\AppData\Local\Temp\tmpj1_y4sn8
12/12/2019 11:12:03 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-vocab.json from cache at C:\Users\nnn\.cache\torch\transformers\f2808208f9bec2320371a9f5f891c184ae0b674ef866b79c58177067d15732dd.1512018be4ba4e8726e41b9145129dc30651ea4fec86aa61f4b9f40bf94eac71
12/12/2019 11:12:03 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-merges.txt from cache at C:\Users\nnn\.cache\torch\transformers\d629f792e430b3c76a1291bb2766b0a047e36fae0588f9dbc1ae51decdff691b.70bec105b4158ed9a1747fea67a43f5dee97855c64d62b6ec3742f4cfdb5feda
12/12/2019 11:12:04 - INFO - transformers.file_utils -   https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-config.json not found in cache or force_download set to True, downloading to C:\Users\nnn\AppData\Local\Temp\tmpyxywrts1
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 176/176 [00:00<00:00, 17738.31B/s]
12/12/2019 11:12:04 - INFO - transformers.file_utils -   copying C:\Users\nnn\AppData\Local\Temp\tmpyxywrts1 to cache at C:\Users\nnn\.cache\torch\transformers\4be02c5697d91738003fb1685c9872f284166aa32e061576bbe6aaeb95649fcf.085d5f6a8e7812ea05ff0e6ed0645ab2e75d80387ad55c1ad9806ee70d272f80
12/12/2019 11:12:04 - INFO - transformers.file_utils -   creating metadata file for C:\Users\nnn\.cache\torch\transformers\4be02c5697d91738003fb1685c9872f284166aa32e061576bbe6aaeb95649fcf.085d5f6a8e7812ea05ff0e6ed0645ab2e75d80387ad55c1ad9806ee70d272f80
12/12/2019 11:12:04 - INFO - transformers.file_utils -   removing temp file C:\Users\nnn\AppData\Local\Temp\tmpyxywrts1
12/12/2019 11:12:04 - INFO - transformers.configuration_utils -   loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-config.json from cache at C:\Users\nnn\.cache\torch\transformers\4be02c5697d91738003fb1685c9872f284166aa32e061576bbe6aaeb95649fcf.085d5f6a8e7812ea05ff0e6ed0645ab2e75d80387ad55c1ad9806ee70d272f80
12/12/2019 11:12:04 - INFO - transformers.configuration_utils -   Model config {
  "attn_pdrop": 0.1,
  "embd_pdrop": 0.1,
  "finetuning_task": null,
  "initializer_range": 0.02,
  "layer_norm_epsilon": 1e-05,
  "n_ctx": 1024,
  "n_embd": 768,
  "n_head": 12,
  "n_layer": 12,
  "n_positions": 1024,
  "num_labels": 1,
  "output_attentions": false,
  "output_hidden_states": false,
  "output_past": true,
  "pruned_heads": {},
  "resid_pdrop": 0.1,
  "summary_activation": null,
  "summary_first_dropout": 0.1,
  "summary_proj_to_labels": true,
  "summary_type": "cls_index",
  "summary_use_proj": true,
  "torchscript": false,
  "use_bfloat16": false,
  "vocab_size": 50257
}

12/12/2019 11:12:04 - INFO - transformers.file_utils -   https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-pytorch_model.bin not found in cache or force_download set to True, downloading to C:\Users\nnn\AppData\Local\Temp\tmpn8i9o_tm
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 548118077/548118077 [01:12<00:00, 7544610.26B/s]
12/12/2019 11:13:18 - INFO - transformers.file_utils -   copying C:\Users\nnn\AppData\Local\Temp\tmpn8i9o_tm to cache at C:\Users\nnn\.cache\torch\transformers\4295d67f022061768f4adc386234dbdb781c814c39662dd1662221c309962c55.778cf36f5c4e5d94c8cd9cefcf2a580c8643570eb327f0d4a1f007fab2acbdf1
12/12/2019 11:13:24 - INFO - transformers.file_utils -   creating metadata file for C:\Users\nnn\.cache\torch\transformers\4295d67f022061768f4adc386234dbdb781c814c39662dd1662221c309962c55.778cf36f5c4e5d94c8cd9cefcf2a580c8643570eb327f0d4a1f007fab2acbdf1
12/12/2019 11:13:24 - INFO - transformers.file_utils -   removing temp file C:\Users\nnn\AppData\Local\Temp\tmpn8i9o_tm
12/12/2019 11:13:24 - INFO - transformers.modeling_utils -   loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-pytorch_model.bin from cache at C:\Users\nnn\.cache\torch\transformers\4295d67f022061768f4adc386234dbdb781c814c39662dd1662221c309962c55.778cf36f5c4e5d94c8cd9cefcf2a580c8643570eb327f0d4a1f007fab2acbdf1
12/12/2019 11:13:32 - INFO - __main__ -   Namespace(device=device(type='cpu'), length=100, model_name_or_path='gpt2', model_type='gpt2', n_gpu=0, no_cuda=False, num_samples=1, padding_text='', prompt='My job is', repetition_penalty=1.0, seed=42, stop_token=None, temperature=1.0, top_k=0, top_p=0.9, xlm_lang='')
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [00:23<00:00,  2.49it/s]
 to know when it will change, it's up to you."

National Communications Director Alex Brynner said the Trump administration needs to help then-Secretary of State Rex Tillerson learn from him.

"The Cabinet, like any other government job, has to be attentive to the needs of an individual that might challenge his or her position," Brynner said. "This is especially true in times of renewed volatility."

Brynner said Tillerson has not "failed at vetting

Output:
Did you read the pre-requisites of the task:

  Installing PyTorch-Transformers on your Machine
  Installing Pytorch-Transformers is pretty straightforward in Python. You can simply use pip install:

     pip install pytorch-transformers

  or if you are working on Colab:

     !pip install pytorch-transformers

  Since most of these models are GPU-heavy, I would suggest working with Google Colab for this part of the article.

Link to Colab: https://colab.research.google.com/notebooks/welcome.ipynb#recent=true

The command syntax used in colab is little bit different, as you can observe from pip command above.

The example command most probably runs better with removal of \ characters (when on cmd) and having all command arguments on same line separated by space char only.

Command may require heavy calculation, so colab with syntax from example directly works in there.

Edit:

Now, looking back your code, there is an important typo:

use prompt with another p, not promt. Then you'd most probably be able to input the seed for the algorithm. I am testing this on my own computer and will soon comment if I find another issue.

Edit2:

My test took some time, now I am ready. I had to follow 3 Readme.md files to get the tutorial to work: one from the root folder, one below tranformers and lastly, the one from samples. Then running this command I got my first results:

  python pytorch-transformers/examples/run_generation.py --model_type=gpt2 --length=2 --model_name_or_path=gpt2 --prompt="My job is"

..and the computer thought a time and said quite well: "to know".


Instruction: How to apply bounds on a variable when performing optimisation in Pytorch?

Input:
I am trying to use Pytorch for non-convex optimisation, trying to maximise my objective (so minimise in SGD). I would like to bound my dependent variable x > 0, and also have the sum of my x values be less than 1000.

I think I have the penalty implemented correctly in the form of a ramp penalty, but am struggling with the bounding of the x variable. In Pytorch you can set the bounds using clamp but it doesn't seem appropriate in this case. I think this is because optim needs the gradients free under the hood. Full working example:

import torch
from torch.autograd import Variable
import numpy as np

def objective(x, a, b, c):   # Want to maximise this quantity (so minimise in SGD)
    d = 1 / (1 + torch.exp(-a * (x)))

    # Checking constraint
    exceeded_limit = constraint(x).item()
    #print(exceeded_limit)

    obj = torch.sum(d * (b * c - x))

    # If overlimit add ramp penalty
    if  exceeded_limit < 0:
        obj = obj - (exceeded_limit * 10)
        print("Exceeded limit")

    return - obj

def constraint(x, limit = 1000): # Must be > 0
    return limit - x.sum()

N = 1000

# x is variable to optimise for
x = Variable(torch.Tensor([1 for ii in range(N)]), requires_grad=True)
a = Variable(torch.Tensor(np.random.uniform(0,100,N)), requires_grad=True)
b = Variable(torch.Tensor(np.random.rand(N)), requires_grad=True)
c = Variable(torch.Tensor(np.random.rand(N)), requires_grad=True)

# Would like to include the clamp
# x = torch.clamp(x, min=0)

# Non-convex methodf
opt = torch.optim.SGD([x], lr=.01)

for i in range(10000):
    # Zeroing gradients
    opt.zero_grad()

    # Evaluating the objective
    obj = objective(x, a, b, c)

    # Calculate gradients
    obj.backward()
    opt.step()
    if i%1000==0:  print("Objective: %.1f" % -obj.item())

print("\nObjective: {}".format(-obj))
print("Limit: {}".format(constraint(x).item()))

if torch.sum(x<0) > 0: print("Bounds not met")
if  constraint(x).item() < 0: print("Constraint not met")

Any suggestions as to how to impose the bounds would be appreciated, either using clamp or otherwise. Or generally advice on non-convex optimisation using Pytorch. This is a much simpler and scaled down version of the problem I'm working so am trying to find a lightweight solution if possible. I am considering using a workaround such as transforming the x variable using an exponential function but then you'd have to scale the function to avoid the positive values becoming infinite, and I want some flexibility with being able to set the constraint.

Output:
I meet the same problem with you.
I want to apply bounds on a variable in PyTorch, too.
And I solved this problem by the below Way3.
Your example is a little compliex but I am still learning English.
So I give a simpler example below.
For example, there is a trainable variable v, its bounds is (-1, 1)
v = torch.tensor((0.5, ）, require_grad=True)
v_loss = xxxx
optimizer.zero_grad()
v_loss.backward()
optimizer.step()

Way1. RuntimeError: a leaf Variable that requires grad has been used in an in-place operation.
v.clamp_(-1, 1)

Way2. RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed.
v = torch.clamp(v, -1, +1)  # equal to v = v.clamp(-1, +1)

Way3. NotError. I solved this problem in Way3.
with torch.no_grad():
    v[:] = v.clamp(-1, +1)  # You must use v[:]=xxx instead of v=xxx


Instruction: Weight Initialization from pretrained BERT error in pytorch

Input:
I am trying to train the model using pretrained model(BERT) using pytorch. The pretrained model weights still arent accepted.

I see this error:

Weights of BertForMultiLable not initialized from pretrained model: ['classifier.weight', 'classifier.bias']
Weights from pretrained model not used in BertForMultiLable: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias']

Here is the complete traceback:

Training/evaluation parameters Namespace(adam_epsilon=1e-08, arch='bert', data_name='ICD9', do_data=False, do_lower_case=True, do_test=False, do_train=True, epochs=6, eval_batch_size=8, eval_max_seq_len=256, fp16=False, fp16_opt_level='O1', grad_clip=1.0, gradient_accumulation_steps=1, learning_rate=2e-05, local_rank=-1, loss_scale=0, mode='min', monitor='valid_loss', n_gpu='0', resume_path='', save_best=True, seed=42, sorted=1, train_batch_size=8, train_max_seq_len=256, valid_size=0.2, warmup_proportion=0.1, weight_decay=0.01)
Loading examples from cached file pybert/dataset/cached_train_examples_bert
Loading features from cached file pybert/dataset/cached_train_features_256_bert
sorted data by th length of input
Loading examples from cached file pybert/dataset/cached_valid_examples_bert
Loading features from cached file pybert/dataset/cached_valid_features_256_bert
initializing model
loading configuration file pybert/pretrain/bert/base-uncased/config.json
Model config {
  "attention_probs_dropout_prob": 0.1,
  "finetuning_task": null,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "is_decoder": false,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "num_labels": 19,
  "output_attentions": false,
  "output_hidden_states": false,
  "output_past": true,
  "pruned_heads": {},
  "torchscript": false,
  "type_vocab_size": 2,
  "use_bfloat16": false,
  "vocab_size": 28996
}

loading weights file pybert/pretrain/bert/base-uncased/pytorch_model.bin
Weights of BertForMultiLable not initialized from pretrained model: ['classifier.weight', 'classifier.bias']
Weights from pretrained model not used in BertForMultiLable: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias']
initializing callbacks
***** Running training *****
  Num examples = 21479
  Num Epochs = 6
  Total train batch size (w. parallel, distributed & accumulation) = 8
  Gradient Accumulation steps = 1
  Total optimization steps = 16110
Warning: There's no GPU available on this machine, training will be performed on CPU.
Warning: The number of GPU's configured to use is 0, but only 0 are available on this machine.
Traceback (most recent call last):
  File "run_bert.py", line 227, in <module>
    main()
  File "run_bert.py", line 220, in main
    run_train(args)
  File "run_bert.py", line 125, in run_train
    trainer.train(train_data=train_dataloader, valid_data=valid_dataloader, seed=args.seed)
  File "/home/aditya_vartak/bert_pytorch/pybert/train/trainer.py", line 168, in train
    summary(self.model,*(input_ids, segment_ids,input_mask),show_input=True)
  File "/home/aditya_vartak/bert_pytorch/pybert/common/tools.py", line 307, in summary
    model(*inputs)
  File "/home/aditya_vartak/virtualenvs/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/aditya_vartak/bert_pytorch/pybert/model/nn/bert_for_multi_label.py", line 14, in forward
    outputs = self.bert(input_ids, token_type_ids=token_type_ids,attention_mask=attention_mask, head_mask=head_mask)
  File "/home/aditya_vartak/virtualenvs/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/aditya_vartak/virtualenvs/anaconda3/envs/pytorch/lib/python3.7/site-packages/transformers/modeling_bert.py", line 722, in forward
    embedding_output = self.embeddings(input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds)
  File "/home/aditya_vartak/virtualenvs/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 533, in __call__
    result = hook(self, input)
  File "/home/aditya_vartak/bert_pytorch/pybert/common/tools.py", line 269, in hook
    summary[m_key]["input_shape"] = list(input[0].size())
IndexError: tuple index out of range

Any help would be great. Thanks in advance

Output:
for the Error you cited, actually I think that is only a Warning that states you  you are loading on your architecture BertForMultiLable the Weights from pretrained model that was not trained for that specific tasks.
Similar warning discussion here
The real error here it's another: IndexError: tuple index out of range. but for that you should attach some code and more information about what you are doing .


Instruction: Text classification with torchnlp

Input:
i'm trying to build a neural network using pytorch-nlp (https://pytorchnlp.readthedocs.io/en/latest/).
My intent is to build a network like this:

Embedding layer (uses pytorch standard layer and from_pretrained method)
Encoder with LSTM (also uses standard nn.LSTM)
Attention mechanism (uses torchnlp.nn.Attention)
Decoder siwth LSTM (as encoder)
Linear layer standard

I'm encountering a major problem with the dimensions of the input sentences (each word is a vector) but most importantly with the attention layer : I don't know how to declare it because i need the exact dimensions of the output from the encoder, but the sequences have varying dimensions (corresponding to the fact that sentences have different number of words).
I've tried to look at torch.nn.utils.rnn.pad_packed_sequence and torch.nn.utils.rnn.pack_padded_sequence since they're supported by LSTM, but i cannot find the solution.
Can anyone help me?
EDIT
I thought about padding all sequences to a specific dimension, but I don't want to truncate longer sequences because I want to keep all the information.

Output: You are on the right track with padding all sequences to a specific dimension. You will have to pick a dimension that is larger than "most" of your sentences but you will need to cutoff some sentences. This blog article should help.


Instruction: PyTorch - to NumPy yields unsized object?

Input:
Converting a PyTorch tensor to NumPy I get

print(nn_result.shape)
# (2433, 2)
np_result = torch.argmax(nn_result).numpy()
type(np_result)
# <type 'numpy.ndarray'>
print(len(np_result))
TypeError: len() of unsized object

Why? I thought per documentation the numpy() function would return a proper ndarray, yet it seems to be incomplete somehow?

Output:
Perhaps you'd want to use torch.argmax(nn_result, dim=1) ? Since dim defaults to 0, it returns just a single number constructed as a tensor. Let me illustrate with the below example:

>>> x = np.array(1)
>>> x.shape
()
>>> len(x)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: len() of unsized object
>>> x = np.array([1])
>>> x.shape
(1,)
>>> len(x)
1

Essentially np.array will take up any object type that you construct with. In the first case, object is not an array because of which you do not see a valid shape. Since it is not an array calling len throws an error.

torch.argmax with dim=0 returns a tensor as seen in the first case of the above example and hence the error.


Instruction: PyTorch vectorise sum lookup quantity into buckets

Input:
Using PyTorch, I have figured out the following code for calculating totals of an item's property by some "bucket index":

DATASET_SIZE = 10
NUM_BUCKETS = 4
bucket_assignment = torch.tensor([0,1,2,3,0,1,2,3,0,1], dtype = torch.long)
values_to_add = torch.tensor([1,2,3,4,5,6,7,8,9,10], dtype = torch.float)
buckets = torch.zeros(NUM_BUCKETS, dtype = torch.float)
buckets.index_add_(0, bucket_assignment, values_to_add)

# Buckets is now tensor([15., 18., 10., 12.])

In my case this is specifically to check allocation bounds on a problem, and later code checks that no bucket is under- or over- allocated.

I would like to check multiple different possible assignments at once (and later pick a best option, code not shown). I thought I could do this by adding another dimension to bucket_assignment plus to buckets and have each row be a different set of assignments. However, this does not work as intended, because the second argument of index_add_ must be a simple vector, I cannot pass in any higher rank tensor.

E.g.

BATCH_SIZE = 2
DATASET_SIZE = 5
NUM_BUCKETS = 3
bucket_assignment = torch.tensor([[0,1,2,0,1], [1,1,1,2,1]], dtype = torch.long)
values_to_add = torch.tensor([1,2,3,4,5], dtype = torch.float)
buckets = torch.zeros(BATCH_SIZE, NUM_BUCKETS, dtype = torch.float)
buckets.index_add_(0, bucket_assignment, values_to_add)

I would like to get this result:

tensor([[5., 7., 3.], [ 0., 11.,  4.]])

Instead, I get an error:

RuntimeError: invalid argument 3: Index is supposed to be a vector at ../aten/src/TH/generic/THTensorEvenMoreMath.cpp:733

That's not unexpected due to the limitations of .index_add, but I don't know how to progress.

I am not sure what other approach would allow me to solve this problem in PyTorch - is there some other torch method I could use that would allow me to achieve the same thing. The main goal here is vectorisation and avoiding loops in Python, as in reality the batch sizes are large and I will be taking advantage of GPU acceleration.

Output:
If the batch size is the problem you could use torch.masked_select to get the values to add up for each bucket torch.masked_select(values_to_add, bucket_assignment == bucket_num), where PyTorch will broadcast the values_to_add and then only iterate over the buckets in plain python like so:

def bucket_sizes(bucket_num):
    mask = bucket_assignment == bucket_num
    buckets = torch.masked_select(values_to_add, mask)
    buckets = torch.split(buckets, list(mask.sum(dim=1)))
    return [bucket.sum() for bucket in buckets]

torch.tensor([bucket_sizes(i) for i in range(NUM_BUCKETS)]).T


Instruction: how to convert mnist images to variables images and labels

Input:
I have a code as below:

dataset = MNIST(path=data_path, download=True, shuffle=True)

if train:
   images, labels = dataset.get_train()
else:
   images, labels = dataset.get_test()

images, labels = images[:n_examples], labels[:n_examples]
images, labels = iter(images.view(-1, 784) / 255), iter(labels)

but when i run it, it gives me this error:

Traceback (most recent call last):
File "C:\Users\Ati\Downloads\Compressed\bindsnet_experiments-
master\experiments\mnist\two_layer_backprop.py", line 135, in <module>
images, labels = dataset.get_train()
AttributeError: 'TorchvisionDatasetWrapper' object has no attribute 'get_train'

I think because get_train() is out of date , it doesn’t support by torchvision
But i tested different ways for converting mnist data to images and labels variables
Who know how could i change it when get_train() doesn’t work
I will appreciate your help if someone helps me on this

Output:
Yes, it looks like class does not exist in the package anymore.

I was able to find the source code for the package you are looking for:

import os
import functools
import operator
import gzip
import struct
import array
import tempfile
try:
    from urllib.request import urlretrieve
except ImportError:
    from urllib import urlretrieve  # py2
try:
    from urllib.parse import urljoin
except ImportError:
    from urlparse import urljoin
import numpy

__version__ = '0.2.2'

# `datasets_url` and `temporary_dir` can be set by the user using:
# >>> mnist.datasets_url = 'http://my.mnist.url'
# >>> mnist.temporary_dir = lambda: '/tmp/mnist'
datasets_url = 'http://yann.lecun.com/exdb/mnist/'
temporary_dir = tempfile.gettempdir

class IdxDecodeError(ValueError):
    """Raised when an invalid idx file is parsed."""
    pass

def download_file(fname, target_dir=None, force=False):
    """Download fname from the datasets_url, and save it to target_dir,
    unless the file already exists, and force is False.

    Parameters
    ----------
    fname : str
        Name of the file to download

    target_dir : str
        Directory where to store the file

    force : bool
        Force downloading the file, if it already exists

    Returns
    -------
    fname : str
        Full path of the downloaded file
    """
    target_dir = target_dir or temporary_dir()
    target_fname = os.path.join(target_dir, fname)

    if force or not os.path.isfile(target_fname):
        url = urljoin(datasets_url, fname)
        urlretrieve(url, target_fname)

    return target_fname

def parse_idx(fd):
    """Parse an IDX file, and return it as a numpy array.

    Parameters
    ----------
    fd : file
        File descriptor of the IDX file to parse

    endian : str
        Byte order of the IDX file. See [1] for available options

    Returns
    -------
    data : numpy.ndarray
        Numpy array with the dimensions and the data in the IDX file

    1. https://docs.python.org/3/library/struct.html
        #byte-order-size-and-alignment
    """
    DATA_TYPES = {0x08: 'B',  # unsigned byte
                  0x09: 'b',  # signed byte
                  0x0b: 'h',  # short (2 bytes)
                  0x0c: 'i',  # int (4 bytes)
                  0x0d: 'f',  # float (4 bytes)
                  0x0e: 'd'}  # double (8 bytes)

    header = fd.read(4)
    if len(header) != 4:
        raise IdxDecodeError('Invalid IDX file, '
                             'file empty or does not contain a full header.')

    zeros, data_type, num_dimensions = struct.unpack('>HBB', header)

    if zeros != 0:
        raise IdxDecodeError('Invalid IDX file, '
                             'file must start with two zero bytes. '
                             'Found 0x%02x' % zeros)

    try:
        data_type = DATA_TYPES[data_type]
    except KeyError:
        raise IdxDecodeError('Unknown data type '
                             '0x%02x in IDX file' % data_type)

    dimension_sizes = struct.unpack('>' + 'I' * num_dimensions,
                                    fd.read(4 * num_dimensions))

    data = array.array(data_type, fd.read())
    data.byteswap()  # looks like array.array reads data as little endian

    expected_items = functools.reduce(operator.mul, dimension_sizes)
    if len(data) != expected_items:
        raise IdxDecodeError('IDX file has wrong number of items. '
                             'Expected: %d. Found: %d' % (expected_items,
                                                          len(data)))

    return numpy.array(data).reshape(dimension_sizes)

def download_and_parse_mnist_file(fname, target_dir=None, force=False):
    """Download the IDX file named fname from the URL specified in dataset_url
    and return it as a numpy array.

    Parameters
    ----------
    fname : str
        File name to download and parse

    target_dir : str
        Directory where to store the file

    force : bool
        Force downloading the file, if it already exists

    Returns
    -------
    data : numpy.ndarray
        Numpy array with the dimensions and the data in the IDX file
    """
    fname = download_file(fname, target_dir=target_dir, force=force)
    fopen = gzip.open if os.path.splitext(fname)[1] == '.gz' else open
    with fopen(fname, 'rb') as fd:
        return parse_idx(fd)

def train_images():
    """Return train images from Yann LeCun MNIST database as a numpy array.
    Download the file, if not already found in the temporary directory of
    the system.

    Returns
    -------
    train_images : numpy.ndarray
        Numpy array with the images in the train MNIST database. The first
        dimension indexes each sample, while the other two index rows and
        columns of the image
    """
    return download_and_parse_mnist_file('train-images-idx3-ubyte.gz')

def test_images():
    """Return test images from Yann LeCun MNIST database as a numpy array.
    Download the file, if not already found in the temporary directory of
    the system.

    Returns
    -------
    test_images : numpy.ndarray
        Numpy array with the images in the train MNIST database. The first
        dimension indexes each sample, while the other two index rows and
        columns of the image
    """
    return download_and_parse_mnist_file('t10k-images-idx3-ubyte.gz')

def train_labels():
    """Return train labels from Yann LeCun MNIST database as a numpy array.
    Download the file, if not already found in the temporary directory of
    the system.

    Returns
    -------
    train_labels : numpy.ndarray
        Numpy array with the labels 0 to 9 in the train MNIST database.
    """
    return download_and_parse_mnist_file('train-labels-idx1-ubyte.gz')

def test_labels():
    """Return test labels from Yann LeCun MNIST database as a numpy array.
    Download the file, if not already found in the temporary directory of
    the system.

    Returns
    -------
    test_labels : numpy.ndarray
        Numpy array with the labels 0 to 9 in the train MNIST database.
    """
    return download_and_parse_mnist_file('t10k-labels-idx1-ubyte.gz')

You can store this in any file, import that file and use the functions as you need (without creating the MNIST object).

Hope this helps. Good luck.


Instruction: Text generation using huggingface's distilbert models

Input:
I've been struggling with huggingface's DistilBERT model for some time now, since the documentation seems very unclear and their examples (e.g. https://github.com/huggingface/transformers/blob/master/notebooks/Comparing-TF-and-PT-models-MLM-NSP.ipynb and https://github.com/huggingface/transformers/tree/master/examples/distillation) are extremely thick and the thing they are showcasing doesn't seem well documented.

I'm wondering if anyone here has any experience and knows of some good code example for basic in-python usage of their models. Namely:

How to properly decode the output of the model into actual text (no matter how I change its shape the tokenizer seems willing to decode it and always yields some sequence of [UNK] tokens)
How to actually use their schedulers+optimizers to train a model for a simple text to text task.

Output:
To decode the output, you can do
        prediction_as_text = tokenizer.decode(output_ids, skip_special_tokens=True)

output_ids contains the generated token ids. It can also be a batch (output ids at every row), then the prediction_as_text will also be a 2D array containing text at every row. skip_special_tokens=True filters out the special tokens used in the training such as  (end of sentence),  (start of sentence), etc. These special tokens vary from model to model of course but almost every model has such special tokens used during training and inference.
There is not an easy way to get rid of unknown tokens[UNK]. The models have limited vocabulary. If a model encounters a subword that is not in its in vocabulary, it is replaced by a special unknown token and the model is trained with these tokens. So, it also learn to generate [UNK]. There are various way to deal with it such as replacing it with the second-highest probable token, or using beam search and taking the most probable sentence that do not contain any unknown tokens. However, if you really want to get rid of these, you should rather use a model that uses Byte Pair Encoding. It solves the problem of unknown words completely. As you can read in this link, Bert and DistilBert uses subwork tokenization and have such a limitation. https://huggingface.co/transformers/tokenizer_summary.html
To use the schedulers and optimizers, you should use the class Trainer and TrainingArguments. Below I posted an example from one of my own projects.
    output_dir=model_directory,
    num_train_epochs=args.epochs,
    per_device_train_batch_size=args.batch_size,
    per_device_eval_batch_size=args.batch_size,
    warmup_steps=500,
    weight_decay=args.weight_decay,
    logging_dir=model_directory,
    logging_steps=100,
    do_eval=True,
    evaluation_strategy='epoch',
    learning_rate=args.learning_rate,
    load_best_model_at_end=True, # the last checkpoint is the best model wrt metric_for_best_model
    metric_for_best_model='eval_loss',
    lr_scheduler_type = 'linear'
    greater_is_better=False,
    save_total_limit=args.epochs if args.save_total_limit == -1 else args.save_total_limit,

)

trainer = Seq2SeqTrainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=val_dataset,
    optimizers=[torch.optim.Adam(params=model.parameters(),
    lr=args.learning_rate), None], // optimizers
    tokenizer=tokenizer,
)

For other scheduler types, see this link: https://huggingface.co/transformers/main_classes/optimizer_schedules.html


Instruction: CTC: blank must be in label range

Input:
summary
I'm adding alphabets to captcha recognition, but pytorch's CTC seems to not working properly when alphabets are added.
What I've tried
At first, I modified BLANK_LABEL to 62 since there are 62 labels(0-9, a-z, A-Z), but it gives me runtime error blank must be in label range. I also tried BLANK_LABEL=0 and then assigning 1~63 as nonblank labels but it outputs NaN as loss.
The code
This is the colab link for the current version of my code: here
below are just core parts of the code.
Constants:
DATASET_PATH = "/home/ik1ne/Downloads/numbers"
MODEL_PATH = "/home/ik1ne/Downloads"

BATCH_SIZE = 50
TRAIN_BATCHES = 180
TEST_BATCHES = 20
TOTAL_BATCHES = TRAIN_BATCHES+TEST_BATCHES
TOTAL_DATASET = BATCH_SIZE*TOTAL_BATCHES

BLANK_LABEL = 63

dataset generation:
!pip install captcha
from captcha.image import ImageCaptcha

import itertools
import os
import random
import string

if not os.path.exists(DATASET_PATH):
    os.makedirs(DATASET_PATH)

characters = "0123456789"+string.ascii_lowercase + string.ascii_uppercase

while(len(list(Path(DATASET_PATH).glob('*'))) < TOTAL_BATCHES):
    captcha_str = "".join(random.choice(characters) for x in range(6))
    if captcha_str in list(Path(DATASET_PATH).glob('*')):
        continue
    ImageCaptcha().write(captcha_str, f"{DATASET_PATH}/{captcha_str}.png")

dataset:
def convert_strseq_to_numseq(s):
    for c in s:
        if c >= '0' and c <= '9':
            return int(c)
        elif c>='a' and c <='z':
            return ord(c)-ord('a')+10
        else:
            return ord(c)-ord('A')+36

class CaptchaDataset(Dataset):
    """CAPTCHA dataset."""

    def __init__(self, root_dir, transform=None):
        self.root_dir = root_dir
        self.image_paths = list(Path(root_dir).glob('*'))
        self.transform = transform

    def __getitem__(self, index):
        image = Image.open(self.image_paths[index])
        if self.transform:
            image = self.transform(image)

        label_sequence = [convert_strseq_to_numseq(c) for c in self.image_paths[index].stem]
        return (image, torch.tensor(label_sequence))

    def __len__(self):
        return len(self.image_paths)

model:
class StackedLSTM(nn.Module):
    def __init__(self, input_size=60, output_size=11, hidden_size=512, num_layers=2):
        super(StackedLSTM, self).__init__()
        self.hidden_size = hidden_size
        self.num_layers = num_layers

        self.dropout = nn.Dropout()
        self.fc = nn.Linear(hidden_size, output_size)
        self.lstm = nn.LSTM(input_size, hidden_size, num_layers)

    def forward(self, inputs, hidden):
        batch_size, seq_len, input_size = inputs.shape
        outputs, hidden = self.lstm(inputs, hidden)
        outputs = self.dropout(outputs)
        outputs = torch.stack([self.fc(outputs[i]) for i in range(width)])
        outputs = F.log_softmax(outputs, dim=2)
        return outputs, hidden

    def init_hidden(self, batch_size):
        weight = next(self.parameters()).data
        return (weight.new(self.num_layers, batch_size, self.hidden_size).zero_(),
                weight.new(self.num_layers, batch_size, self.hidden_size).zero_())

net = StackedLSTM().to(device)

training:
net.train()  # set network to training phase

epochs = 30

# for each pass of the training dataset
for epoch in range(epochs):
    train_loss, train_correct, train_total = 0, 0, 0

    h = net.init_hidden(BATCH_SIZE)

    # for each batch of training examples
    for batch_index, (inputs, targets) in enumerate(train_dataloader):
        inputs, targets = inputs.to(device), targets.to(device)
        h = tuple([each.data for each in h])

        BATCH_SIZE, channels, height, width = inputs.shape

        # reshape inputs: NxCxHxW -> WxNx(HxC)
        inputs = (inputs
                  .permute(3, 0, 2, 1)
                  .contiguous()
                  .view((width, BATCH_SIZE, -1)))

        optimizer.zero_grad()  # zero the parameter gradients
        outputs, h = net(inputs, h)  # forward pass

        # compare output with ground truth
        input_lengths = torch.IntTensor(BATCH_SIZE).fill_(width)
        target_lengths = torch.IntTensor([len(t) for t in targets])
        loss = criterion(outputs, targets, input_lengths, target_lengths)

        loss.backward()  # backpropagation
        nn.utils.clip_grad_norm_(net.parameters(), 10)  # clip gradients
        optimizer.step()  # update network weights

        # record statistics
        prob, max_index = torch.max(outputs, dim=2)
        train_loss += loss.item()
        train_total += len(targets)

        for i in range(BATCH_SIZE):
            raw_pred = list(max_index[:, i].cpu().numpy())
            pred = [c for c, _ in groupby(raw_pred) if c != BLANK_LABEL]
            target = list(targets[i].cpu().numpy())
            if pred == target:
                train_correct += 1

        # print statistics every 10 batches
        if (batch_index + 1) % 10 == 0:
            print(f'Epoch {epoch + 1}/{epochs}, ' +
                  f'Batch {batch_index + 1}/{len(train_dataloader)}, ' +
                  f'Train Loss: {(train_loss/1):.5f}, ' +
                  f'Train Accuracy: {(train_correct/train_total):.5f}')

            train_loss, train_correct, train_total = 0, 0, 0

Output:
This error will occur when the index of blank is larger than the total number of classes, which equals number of chars + blank. What's more, the index starts from 0, instead of 1, so if you have 62 characters in total, their index should be 0-61 and the index of blank should be 62 instead of 63. (Or you can set blank as 0, other characters from 1-62)
You should also check the shape of the output tensor, it should has shape [T, B, C], where T is the time step length, B is the batch size, C is the class num, remember to add blank in to the class num or you will meet the problem


Instruction: Why does torch.from_numpy require a different byte ordering while matplotlib doesn't?

Input:
Here is a piece of code (running on Linux CentOS 7.7.1908, x86_64)

import torch    #v1.3.0
import numpy as np  #v1.14.3
import matplotlib.pyplot as plt
from astropy.io.fits import getdata   #v3.0.2
data, hdr = getdata("afile.fits", 0, header=True) #gives dtype=float32 2d array
plt.imshow(data)
plt.show()

This gives a nice 512x512 image

Now, I would like to convert "data" into a PyTorch tensor:

a = torch.from_numpy(data)

Although, PyTorch raises:

  ValueError: given numpy array has byte order different from the native
  byte order. Conversion between byte orders is currently not supported.

Well, I have tried different manipulations with no success: ie. byteswap(), copy()

An idea?

PS: the same error occurs when I transfer my data to Mac OSX (Mojave) while still matplotlib is ok.

Output:
FITS stores data in big-endian byte ordering (at the time FITS was developed this was a more common machine architecture; sadly the standard has never been updated to allow flexibility on this, although it could easily be done with a single header keyword to indicate endianness of the data...)

According to the Numpy docs Numpy arrays report the endianness of the underlying data as part of its dtype (e.g. a dtype of '>i' means big-endian ints, and 'and change the array's dtype to reflect the new byte order.

Your solution of calling .astype(np.float32) should work, but that's because the np.float32 dtype is explicitly little-endian, so .astype(...) copies an existing array and converts the data in that array, if necessary, to match that dtype. I just wanted to explain exactly why that works, since it might be otherwise unclear why you're doing that.

As for matplotlib it doesn't really have much to do with your question. Numpy arrays can transparently perform operations on data that does not match the endianness of your machine architecture, by automatically performing byte swaps as necessary. Matplotlib and many other scientific Python libraries work directly with Numpy arrays and thus automatically benefit from its transparent handling of endianness.

It just happens that PyTorch (in part because of its very high performance and GPU-centric data handling model) requires you to hand it data that's already in little-endian order, perhaps just to avoid ambiguity. But that's particular to PyTorch and is not specifically a contrast with matplotlib.


Instruction: Using Pytorch's dataloaders & transforms with sklearn

Input:
I have been using pytorch a lot and got used to their dataloaders and transforms, in particular when it comes to data augmentation, as they're very user-friendly and easy to understand.

However, I need to run some ML models from sklearn.

Is there a way to use pytorch's dataloaders for sklearn ?

Output:
Yes, you can. You can do this with sklearn's partial_fit method. Read HERE.

  6.1.3. Incremental learning

  Finally, for 3. we have a number of options inside scikit-learn. Although all algorithms cannot learn
  incrementally (i.e. without seeing all the instances at once), all
  estimators implementing the partial_fit API are candidates. Actually,
  the ability to learn incrementally from a mini-batch of instances
  (sometimes called “online learning”) is key to out-of-core learning as
  it guarantees that at any given time there will be only a small amount
  of instances in the main memory. Choosing a good size for the
  mini-batch that balances relevancy and memory footprint could involve
  some tuning [1].

Not all algorithms can do this, however.

Then, you can use pytorch's dataloader to preprocess the data and feed it in batches to partial_fit.


Instruction: State persistence in shared LSTM layers in Keras

Input:
I am trying to use a shared LSTM layer with state in a Keras model, but it seems that the internal state is modified by each parallel use. This raises two questions:

When training a model with a shared LSTM layer and using stateful=True, are the parallel uses updating the same state also during training?
If my observation is valid, is there a way to use weight-sharing LSTMs such that the state is stored independently for each of the parallel uses?

The code below exemplifies the problem with three sequences sharing the LSTM. The prediction of a full input is compared with the result from splitting the prediction input into two halves and feeding them into the network consecutively.

What can be observed, is that the a1 is the same as the first half of aFull, meaning that the the uses of the LSTM really are in parallel with independent states during the first prediction. I.e., z1 is not affected by the parallel call creating z2 and z3. But a2 is different from the second half of aFull, so there is some interaction between the states of the parallel uses.

What I was hoping is that the concatenation of the two pieces a1 and a2 would be the same as the result from calling the prediction with a longer input sequence, but this doesn't seem to be the case. A further concern is that when this kind of interaction takes place in the prediction, is it also happening during the training.

import keras
import keras.backend as K
import numpy as np

nOut = 3
xShape = (3, 50, 4)
inShape = (xShape[0], None, xShape[2])
batchInShape = (1, ) + inShape
x = np.random.randn(*xShape)

# construct network
xIn = keras.layers.Input(shape=inShape, batch_shape=batchInShape)

# shared LSTM layer
sharedLSTM = keras.layers.LSTM(units=nOut, stateful=True, return_sequences=True, return_state=False)

# split the input on the first axis
x1 = keras.layers.Lambda(lambda x: x[:,0,:,:])(xIn)
x2 = keras.layers.Lambda(lambda x: x[:,1,:,:])(xIn)
x3 = keras.layers.Lambda(lambda x: x[:,2,:,:])(xIn)

# pass each input through the LSTM
z1 = sharedLSTM(x1)
z2 = sharedLSTM(x2)
z3 = sharedLSTM(x3)

# add a singleton dimension
y1 = keras.layers.Lambda(lambda x: K.expand_dims(x, axis=1))(z1)
y2 = keras.layers.Lambda(lambda x: K.expand_dims(x, axis=1))(z2)
y3 = keras.layers.Lambda(lambda x: K.expand_dims(x, axis=1))(z3)

# combine the outputs
y = keras.layers.Concatenate(axis=1)([y1, y2, y3])

model = keras.models.Model(inputs=xIn, outputs=y)
model.compile(loss='mse', optimizer='adam')
model.summary()

# no need to train, since we're interested only what is happening mechanically

# reset to a known state and predict for full input
model.reset_states()
aFull = model.predict(x[np.newaxis,:,:,:])

# reset to a known state and predict for the same input, but in two pieces
model.reset_states()
a1 = model.predict(x[np.newaxis,:,:xShape[1]//2,:])
a2 = model.predict(x[np.newaxis,:,xShape[1]//2:,:])
# combine the pieces
aSplit = np.concatenate((a1, a2), axis=2)

print('full diff: {}, first half diff: {}, second half diff: {}'.format(str(np.sum(np.abs(aFull - aSplit))), str(np.sum(np.abs(aFull[:,:,:xShape[1]//2,:] - aSplit[:,:,:xShape[1]//2,:]))), str(np.sum(np.abs(aFull[:,:,xShape[1]//2:,:] - aSplit[:,:,xShape[1]//2:,:])))))

Update: The behaviour described above was observed with Keras using Tensorflow 1.14 and 1.15 as the backend. Running the same code with tf2.0 (with the adjusted imports) changes the result so that a1 is no longer the same as the first half of aFull. This can be still accomplished by setting stateful=False in the layer instantiation.

This would suggest to me that the way I'm trying to use the recursive layer with shared parameters, but own states for parallel uses, is not really possible like this.

Update 2: It seems that the same functionality has been missed by also other earlier: closed, unanswered question at Keras' github.

For a comparison, here is a scribbling in pytorch (the first time I've tried to use it) implementing a simple network with N parallel LSTMs sharing the weights, but having independent states. In this case the states are stored explicitly in a list and provided to the LSTM cell manually.

import torch
import numpy as np

class sharedLSTM(torch.nn.Module):

    def __init__(self, batchSz, nBands, nDims, outDim):
        super(sharedLSTM, self).__init__()
        self.internalLSTM = torch.nn.LSTM(input_size=nDims, hidden_size=outDim, num_layers=1, bias=True, batch_first=True)
        allStates = list()
        for bandIdx in range(nBands):
            h_0 = torch.zeros(1, batchSz, outDim)
            c_0 = torch.zeros(1, batchSz, outDim)
            allStates.append((h_0, c_0))

        self.allStates = allStates
        self.nBands = nBands

    def forward(self, x):
        allOut = list()
        for dimIdx in range(self.nBands):
            thisSlice = x[:,dimIdx,:,:] # (batchSz, nSteps, nFeats)
            thisState = self.allStates[dimIdx]

            thisY, thisState = self.internalLSTM(thisSlice, thisState)
            self.allStates[dimIdx] = thisState
            allOut.append(thisY[:,None,:,:]) # => (batchSz, 1, nSteps, nFeats)

        y = torch.cat(allOut, dim=1) # => (batchSz, nDims, nSteps, nFeats)

        return y

    def resetStates(self):
        for bandIdx in range(nBands):
            self.allStates[bandIdx][0][:] = 0.0
            self.allStates[bandIdx][1][:] = 0.0

batchSz = 5
nBands = 3
nFeats = 4
nOutDims = 2
net = sharedLSTM(batchSz, nBands, nFeats, nOutDims)
net = net.float()
print(net)

N = 20
x = torch.from_numpy(np.random.rand(batchSz, nBands, N, nFeats)).float()
x1 = x[:, :, :N//2, :]
x2 = x[:, :, N//2:, :]

aa = net.forward(x)
net.resetStates()
a1 = net.forward(x1)
a2 = net.forward(x2)

print('(with reset) first half abs diff: {}'.format(str(torch.sum(torch.abs(a1 - aa[:,:,:N//2,:])).detach().numpy())))
print('(with reset) second half abs diff: {}'.format(str(torch.sum(torch.abs(a2 - aa[:,:,N//2:,:])).detach().numpy())))

Result: the output is the same regardless if we do the prediction in one go or in pieces.

I've tried to replicate this in Keras using sub-classing, but without success:

import keras
import numpy as np

class sharedLSTM(keras.Model):
    def __init__(self, batchSz, nBands, nDims, outDim):
        super(sharedLSTM, self).__init__()
        self.internalLSTM = keras.layers.LSTM(units=outDim, stateful=True, return_sequences=True, return_state=True)
        self.internalLSTM.build((batchSz, None, nDims))
        self.internalLSTM.reset_states()
        allStates = list()
        allSlicers = list()
        for bandIdx in range(nBands):
            allStates.append(None)
            allSlicers.append(keras.layers.Lambda(lambda x, b: x[:, :, b, :], arguments = {'b' : bandIdx}))

        self.allStates = allStates
        self.allSlicers = allSlicers
        self.Concat = keras.layers.Lambda(lambda x: keras.backend.concatenate(x, axis=2))

        self.nBands = nBands

    def call(self, x):
        allOut = list()
        for bandIdx in range(self.nBands):
            thisSlice = self.allSlicers[bandIdx]( x )
            thisState = self.allStates[bandIdx]

            thisY, *thisState = self.internalLSTM(thisSlice, initial_state=thisState)
            self.allStates[bandIdx] = thisState.copy()
            allOut.append(thisY[:,:,None,:])

        y = self.Concat( allOut )
        return y

batchSz = 1
nBands = 3
nFeats = 4
nOutDims = 2
N = 20

model = sharedLSTM(batchSz, nBands, nFeats, nOutDims)
model.compile(optimizer='SGD', loss='mae')

x = np.random.rand(batchSz, N, nBands, nFeats)
x1 = x[:, :N//2, :, :]
x2 = x[:, N//2:, :, :]

aa = model.predict(x)

model.reset_states()
a1 = model.predict(x1)
a2 = model.predict(x2)

print('(with reset) first half abs diff: {}'.format(str(np.sum(np.abs(a1 - aa[:,:N//2,:,:])))))
print('(with reset) second half abs diff: {}'.format(str(np.sum(np.abs(a2 - aa[:,N//2:,:,:])))))

If you now ask "why don't you then use torch and shut up?", the answer is that the surrounding experimental framework has been built assuming Keras and changing it would be a non-negligible amount of work.

Output:
Based on my current understanding of the behaviour of LSTMs (and other RNNs) in Keras is that using a shared LSTM layer in a stateful=True mode does not work as one would expect and there is only one state variable that gets updated through all the parallel uses. So the answers to the questions appear to be:

Yes, they are. The processing runs over one of the many parallel sequences, stores the state at the end, and uses this as the initial state for the second parallel sequence, and so forth.
Yes, but it requires some work. See below for the details.

I've managed to accomplish handling the states in two ways. First is deriving sub-classes from Keras' LSTM and LSTMCell, and overloading LSTMCell.call() to handle the parallel data streams by splitting the input, and storing and recovering the state of each parallel stream. A drawback here is that the input shape to an RNN is fixed to be 3D, which means that the parallel inputs need to be reshaped into the feature dimension along with the real features.

The second approach is to create a wrapper Layer not completely dissimilar to the sharedLSTM-Model in the question, containing slicing of the input to parallel streams, calling the internal LSTM with the correct state for each stream, and storing the returned states. The state storage update in the list works through add_update() call inserted into the end of call(). This add_update() does not (seem to) work with Model, hence Layer. However, when run with Keras <2.3, the weights of the nested layers are not tracked or updated, so Keras 2.3+ or TF2 is needed.


Instruction: PyTorch: initializing weight with numpy array + create a constant tensor

Input:
I have the following code :

self.wi = nn.Embedding(num_embeddings, embedding_dim)
self.wj = nn.Embedding(num_embeddings1, embedding_dim)
self.bi = nn.Embedding(num_embeddings, 1)
self.bj = nn.Embedding(num_embeddings1, 1)

self.wi.weight.data.uniform_(-1, 1)
self.wj.weight.data.uniform_(-1, 1)
self.bi.weight.data.zero_()
self.bj.weight.data.zero_()

I want to initialize the weights with numpy array, and I want to create a constant tensor, which is also a numpy array.
I am new to PyTorch, and I appreciate any help.

Output:
You can initialize embedding layers with the function nn.Embedding.from_pretrained().

In your specific case, you would still have to firstly convert the numpy.array to a torch.Tensor, but otherwise it is very straightforward:

import torch as t
import torch.nn as nn
import numpy as np

# This can be whatever initialization you want to have
init_array = np.zeros([num_embeddings, embedding_dims])

# As @Daniel Marchand mentioned in his answer,
# you do have to cast it explicitly as a tensor, otherwise it won't work.
wi = nn.Embedding.from_pretrained(t.tensor(init_array), freeze=False)

The parameter freeze=False is important if you still want to train your network afterwards, as otherwise you would keep the embeddings at the same constant values.
Generally, .from_pretrained is used to "transfer" learned embeddings, but it does work for your case, too.


Instruction: Using tensorboard in pytorch, but get blank page?

Input:
I am using tensorboard in pytorch 1.3.1, and I did exactly the same in the pytorch docs for tensorboard.
After running tensorboard --logdir=runs, I got this enter image description here.

$ tensorboard --logdir=runs
TensorFlow installation not found - running with reduced feature set.
Serving TensorBoard on localhost; to expose to the network, use a proxy or pass --bind_all
TensorBoard 2.1.0 at http://localhost:6006/ (Press CTRL+C to quit)

And after opening http://localhost:6006/, I got blank page like this

I also tried tensorboardX, and got the same result. Could you please tell how to solve the problem? thx.

Output: I am using Torch 1.4.0 on Windows and I had the same issue. Turns out I had installed the 2.x version of Tensorboard. I reverted back to 1.15.0 and it solved the issue.


Instruction: How to convert Conv1d into Conv2d? [PyTorch]

Input:
Is it possible with PyTorch to use Conv2d to perform a Conv1d? The question may seem weird, but I need to use a tool that is not compatible with conv1d, but it works with conv2d.

What if I have Conv1d(in,out, kernel_size=3, stride=stride, padding=1, bias=False)?
May unsqueeze help me?

I have the same problem with AvgPool1d (-> AvgPool2d) and MaxPool1d (-> MaxPool2d).

Output:
A Conv2D is mostly a generalized version of Conv1D. You can of course use a degenerate version of Conv2D to reproduce a 1D convolution -
You'll need to add in another dimension to the data:

data_pnt = data_pnt [..., numpy.newaxis]

You'll also need to specify the kernel size - You'll be choosing a 1D kernel - for example:

Conv2d(in,out, kernel_size=(3,1), <Other Parameters>)


Instruction: Is Pytorch DataLoader Iteration order stable?

Input:
Is the iteration order for a Pytorch Dataloader guaranteed to be the same (under mild conditions)?

For instance:

dataloader = DataLoader(my_dataset, batch_size=4,
                        shuffle=True, num_workers=4)
print("run 1")
for batch in dataloader:
  print(batch["index"])

print("run 2")
for batch in dataloader:
  print(batch["index"])

So far, I've tried testing it and it appears to not be fixed, same order for both runs. Is there a way to make the order the same? Thanks

edit: i have also tried doing

unlabeled_sampler = data.sampler.SubsetRandomSampler(unlabeled_indices)
unlabeled_dataloader = data.DataLoader(train_dataset,
                sampler=unlabeled_sampler, batch_size=args.batch_size, drop_last=False)

and then iterating through the dataloader twice, but the same non-determinism results.

Output:
The short answer is no, when shuffle=True the iteration order of a DataLoader isn't stable between iterations. Each time you iterate on your loader the internal RandomSampler creates a new random order.

One way to get a stable shuffled DataLoader is to create a Subset dataset using a shuffled set of indices.

shuffled_dataset = torch.utils.data.Subset(my_dataset, torch.randperm(len(my_dataset)).tolist())
dataloader = DataLoader(shuffled_dataset, batch_size=4, num_workers=4, shuffled=False)


Instruction: install conda on windows with conda or pip

Input:
I am trying to install pytorch on my window.
First, I get command conda install pytorch torchvision cpuonly -c pytorchfrom here
(PyTorch Build:Stable(1.3);Your OS:Windows;Package:Conda;Language:Python3.6;CUDA:None), there are some problems described as followings:
**(python36) C:\Users\li_dan0109>conda install pytorch torchvision cpuonly -c pytorch
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
PackagesNotFoundError: The following packages are not available from current channels:
  - pytorch
Current channels:
  - https://conda.anaconda.org/pytorch/win-32
  - https://conda.anaconda.org/pytorch/noarch
  - https://repo.anaconda.com/pkgs/main/win-32
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/r/win-32
  - https://repo.anaconda.com/pkgs/r/noarch
  - https://repo.anaconda.com/pkgs/msys2/win-32
  - https://repo.anaconda.com/pkgs/msys2/noarch
To search for alternate channels that may provide the conda package you're
looking for, navigate to
    https://anaconda.org
and use the search bar at the top of the page.**

Then I also switch pip install by the command pip3 install torch==1.3.1+cpu torchvision==0.4.2+cpu -f https://download.pytorch.org/whl/torch_stable.html Then I met this error:
**(python36) C:\Users\li_dan0109>pip3 install torch==1.3.1+cpu torchvision==0.4.2+cpu -f https://download.pytorch.org/whl/torch_stable.html
Looking in links: https://download.pytorch.org/whl/torch_stable.html
ERROR: Could not find a version that satisfies the requirement torch==1.3.1+cpu (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
ERROR: No matching distribution found for torch==1.3.1+cpu**

I really don't know how to fix it, please help me and thank you very much!

Output:
It looks as though you may have the 32-bit installation of Python, in which case you're issue is this: #16633.

Just be aware, that pyTorch doesn't work on 32-bit systems.

Please use Windows and Python 64-bit version.


Instruction: Pytorch Deep Learning - Class Model() and training function

Input:
I am new to Pytorch and I am going through this tutorial to figure out how to do deep learning with this library. I have problem figuring out part of the code.

There is a class called Net and an object called model instantiated from it. Then there is the training function called  train(epoch). In the next line in the train function body I see this: model.train() which I can't make any sense of. Would you please help me in understanding this part of the code? how we are calling a method of a class when this method has not been defined inside the class? and how come the method has the exact same name as function it has been called inside?    This is the class definition:

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 32, kernel_size=3)
        self.conv2 = nn.Conv2d(32, 64, kernel_size=3)
        self.conv2_drop = nn.Dropout2d()
        self.fc1 = nn.Linear(2304, 256)
        self.fc2 = nn.Linear(256, 17)

    def forward(self, x):
        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
        x = x.view(x.size(0), -1) # Flatten layer
        x = F.relu(self.fc1(x))
        x = F.dropout(x, training=self.training)
        x = self.fc2(x)
        return F.sigmoid(x)

model = Net() # On CPU
# model = Net().cuda() # On GPU

And this is the function defined after this class:

def train(epoch):
    model.train()
    for batch_idx, (data, target) in enumerate(train_loader):
        # data, target = data.cuda(async=True), target.cuda(async=True) # On GPU
        data, target = Variable(data), Variable(target)
        optimizer.zero_grad()
        output = model(data)
        loss = F.binary_cross_entropy(output, target)
        loss.backward()
        optimizer.step()
        if batch_idx % 10 == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                epoch, batch_idx * len(data), len(train_loader.dataset),
                100. * batch_idx / len(train_loader), loss.data[0]))

Output:
train vs model.train
The def train(epochs): ... is the method to train the model and is not an attribute of Net class. model is an object of Net class, that inherits from nn.Module.
In PyTorch, all layers inherit from nn.Module and that gives them a lot of common functionality like model.children() or layer.children(), model.apply(), etc.
The model.train() is similarly implemented in nn.Module. It doesn't actually train the model or any layer that inherits from nn.Module but sets it in train mode, so say it's equivalent to doing model.set_train().
There is an equivalent method model.eval() you would call in to set the evaluation mode before testing the model.
Why do you even need it?
There can be some parameters in a layer/model that are suppose to act differently while training and evaluation modes. The obvious example are BatchNorm's γ and β


Instruction: Using pretrained models in Pytorch for Semantic Segmentation, then training only the fully connected layers with our own dataset

Input:
I am learning Pytorch and trying to understand how the library works for semantic segmentation.
What I've understood so far is that we can use a pre-trained model in pytorch. I've found an article which was using this model in the .eval() mode but I have not been able to find any tutorial on using such a model for training on our own dataset. I have a very small dataset and I need transfer learning to get results. My goal is to only train the FC layers with my own data. How is that achievable in Pytorch without complicating the code with OOP or so many .py files. I have been having a hard time figuring out such repos in github as I am not the most proficient person when it comes to OOP. I have been using Keras for Deep Learning until recently and there everything is easy and straightforward. Do I have the same options in Pycharm?
I appreciate any guidance on this. I need to run a piece of code that does the semantic segmentation and I am really confused about many of the steps I need to take.

Output:
Assume you start with a pretrained model called model. All of this occurs before you pass the model any data.

You want to find the layers you want to train by looking at all of them and then indexing them using model.children(). Running this command will show you all of the blocks and layers.

list(model.children())

Suppose you have now found the layers that you want to finetune (your FC layers as you describe). If the layers you want to train are the last 5 you can grab all of the layers except for the last 5 in order to set their requires_grad params to False so they don't train when you run the training algorithm.

list(model.children())[-5:]

Remove those layers:

layer_list = list(model.children())[-5:]

Rebuild model using sequential:

model_small = nn.Sequential(*list(model.children())[:-5])

Set requires_grad params to False:

for param in model_small.parameters():
    param.requires_grad = False

Now you have a model called model_small that has all of the layers except the layers you want to train. Now you can reattach the layers that your removed and they will intrinsically have the requires_grad param set to True. Now when you train the model it will only update the weights on those layers.

model_small.avgpool_1 = nn.AdaptiveAvgPool2d()
model_small.lin1 = nn.Linear()
model_small.logits = nn.Linear()
model_small.softmax = nn.Softmax()

model = model_small.to(device)


Instruction: Custom backward/optimization steps in pytorch-lightning

Input:
I would like to implement the training loop below in pytorch-lightning (to be read as pseudo-code). The peculiarity is that the backward and optimization steps are not performed for every batch.

(Background: I am trying to implement a few-shots learning algorithm; although I need to make predictions at every step -- forward method
-- I need to perform the gradient updates at random -- if- block.

for batch in batches:
    x, y = batch
    loss = forward(x,y)

    optimizer.zero_grad()

    if np.random.rand() > 0.5:
        loss.backward()
        optimizer.step()

My proposed solution entails implementing the backward and the optimizer_step methods as follows:

def backward(self, use_amp, loss, optimizer):
        self.compute_grads = False
        if np.random.rand() > 0.5:
            loss.backward()
            nn.utils.clip_grad_value_(self.enc.parameters(), 1)
            nn.utils.clip_grad_value_(self.dec.parameters(), 1)
            self.compute_grads = True
        return

    def optimizer_step(self, current_epoch, batch_nb, optimizer, optimizer_i, second_order_closure=None):
        if self.compute_grads:
            optimizer.step()
            optimizer.zero_grad()
        return

Note: In this way I need to store a compute_grads attribute at the class level.

What is the "best-practice" way to implement it in pytorch-lightning? Is there a better way to use the hooks?

Output:
This is a good way to do it! that's what the hooks are for.

There is a new Callbacks module that might also be helpful:
https://pytorch-lightning.readthedocs.io/en/0.7.1/callbacks.html


Instruction: RuntimeError: cuda runtime error (710) : device-side assert triggered at

Input:
  Traning the image classification with pytorch  got following
  error messageK

  RuntimeError                              Traceback (most recent call
  last)  in
       29         print(len(train_loader.dataset),len(valid_loader.dataset))
       30         #break
  ---> 31         train_loss, train_acc ,model= train(model, device, train_loader, optimizer, criterion)
       32         valid_loss, valid_acc,model = evaluate(model, device, valid_loader, criterion)
       33

   in train(model, device, iterator,
  optimizer, criterion)
       21         acc = calculate_accuracy(fx, y)
       22         #print("5.")
  ---> 23         loss.backward()
       24
       25         optimizer.step()

  ~/venv/lib/python3.7/site-packages/torch/tensor.py in backward(self,
  gradient, retain_graph, create_graph)
      164                 products. Defaults to False.
      165         """
  --> 166         torch.autograd.backward(self, gradient, retain_graph, create_graph)
      167
      168     def register_hook(self, hook):

  ~/venv/lib/python3.7/site-packages/torch/autograd/init.py in
  backward(tensors, grad_tensors, retain_graph, create_graph,
  grad_variables)
       97     Variable._execution_engine.run_backward(
       98         tensors, grad_tensors, retain_graph, create_graph,
  ---> 99         allow_unreachable=True)  # allow_unreachable flag
      100
      101

  RuntimeError: cuda runtime error (710) : device-side assert triggered
  at /pytorch/aten/src/THC/generic/THCTensorMath.cu:26

Related code block is here

def train(model, device, iterator, optimizer, criterion):

print('train')
epoch_loss = 0
epoch_acc = 0

model.train()

for (x, y) in iterator:
    #print(x,y)
    x,y = x.cuda(), y.cuda()
    #x = x.to(device)
    #y = y.to(device)
    #print('1')
    optimizer.zero_grad()
    #print('2')
    fx = model(x)
    #print('3')
    loss = criterion(fx, y)
    #print("4.loss->",loss)
    acc = calculate_accuracy(fx, y)
    #print("5.")
    loss.backward()

    optimizer.step()

    epoch_loss += loss.item()
    epoch_acc += acc.item()

return epoch_loss / len(iterator), epoch_acc / len(iterator),model

    EPOCHS = 5
    SAVE_DIR = 'models'
    MODEL_SAVE_PATH = os.path.join(SAVE_DIR, 'please.pt')
    from torch.utils.data import DataLoader
    best_valid_loss = float('inf')

    if not os.path.isdir(f'{SAVE_DIR}'):
        os.makedirs(f'{SAVE_DIR}')
    print("start")
    for epoch in range(EPOCHS):
        print('================================',epoch ,'================================')
        for i , (train_idx, valid_idx) in enumerate(zip(train_indexes, valid_indexes)):
            print(i,train_idx,valid_idx,len(train_idx),len(valid_idx))

            traindf = df_train.iloc[train_index, :].reset_index()
            validdf = df_train.iloc[valid_index, :].reset_index()

            #traindf = df_train
            #validdf = df_train

            train_dataset = TrainDataset(traindf, mode='train', transforms=data_transforms)
            valid_dataset = TrainDataset(validdf, mode='valid', transforms=data_transforms)

            train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
            valid_loader = DataLoader(valid_dataset, batch_size=batch_size, shuffle=False)

            print(len(train_loader.dataset),len(valid_loader.dataset))
            #break
            train_loss, train_acc ,model= train(model, device, train_loader, optimizer, criterion)
            valid_loss, valid_acc,model = evaluate(model, device, valid_loader, criterion)

            if valid_loss < best_valid_loss:
                best_valid_loss = valid_loss
                torch.save(model,MODEL_SAVE_PATH)

            print(f'| Epoch: {epoch+1:02} | Train Loss: {train_loss:.3f} | Train Acc: {train_acc*100:05.2f}% | Val. Loss: {valid_loss:.3f} | Val. Acc: {valid_acc*100:05.2f}% |')

  splits = zip(train_indexes, valid_indexes)
  [ 3692  3696  3703 ... 30733 30734 30735] [   0    1    2 ... 4028
  4041 4046] [    0     1     2 ... 30733 30734 30735] [3692 3696 3703
  ... 7986 7991 8005] [    0     1     2 ... 30733 30734 30735] [ 7499
  7500  7502 ... 11856 11858 11860] [    0     1     2 ... 30733 30734
  30735] [11239 11274 11280 ... 15711 15716 15720] [    0     1     2
  ... 30733 30734 30735] [15045 15051 15053 ... 19448 19460 19474] [
  0     1     2 ... 30733 30734 30735] [18919 18920 18926 ... 23392
  23400 23402] [    0     1     2 ... 30733 30734 30735] [22831 22835
  22846 ... 27118 27120 27124] [    0     1     2 ... 27118 27120 27124]
  [26718 26721 26728 ... 30733 30734 30735]

Output:
What was your loss function?
I got this error too. My problem was a multi-class classification and I was using a crossEntropy loss.
As it say in the documentations, labels should be in the range [0, C-1] where C is number of classes.
But my labels were not in the range and when I used proper values for labels, Everything was ok.


Instruction: Why Keras behave better than Pytorch under the same network configuration?

Input:
Recently,  I have compared unet++ implementation of Keras version and Pytorch version on the same dataset. However, with Keras the loss decrease continuously and the accuracy is higher after 10 epochs, while with Pytorch the loss decrease unevenly and the  accuracy is lower after 10 epochs. Anyone has met such problems and has any answers?
the final pytorch training process is like:

2019-12-15 18:14:20 Epoch:9 Iter: 1214/1219  loss:0.464673 acc:0.581713
2019-12-15 18:14:21 Epoch:9 Iter: 1215/1219  loss:0.450462 acc:0.584101
2019-12-15 18:14:21 Epoch:9 Iter: 1216/1219  loss:0.744811 acc:0.293406
2019-12-15 18:14:22 Epoch:9 Iter: 1217/1219  loss:0.387612 acc:0.735630
2019-12-15 18:14:23 Epoch:9 Iter: 1218/1219  loss:0.767146 acc:0.364759

the final keras training process is like:

685/690 [============================>.] - ETA: 2s - loss: 0.4940 - acc:  0.7309
686/690 [============================>.] - ETA: 1s - loss: 0.4941 - acc: 0.7306
687/690 [============================>.] - ETA: 1s - loss: 0.4939 - acc:  0.7308
688/690 [============================>.] - ETA: 0s - loss: 0.4942 - acc:  0.7303
689/690 [============================>.] - ETA: 0s - loss: 0.4943 - acc: 0.7302

Output:
Well, it's pretty hard to say without any code snippets. that being said, in general, initialization is way more important than you might think. I'm sure that the default initialization of pytorch is different from keras and I had similar issues in the past.

Another thing to check is the optimizer parameters, make sure that not only you are using the same optimizer(sgd, adam, ...) but also with the same parameters(lr, beta, momentum, ...)


Instruction: Is there any diffrence between index_select and tensor[sequence] in PyTorch?

Input:
everyone. I'm new to PyTorch. Now I'm learning the indexing of a tensor. I notice that we can indexing a tensor by tensor.index_select() and tensor[sequence].

In [1]: x = torch.randn(3, 4)

In [2]: indices = torch.tensor([0, 2])

In [3]: x.index_select(0, indices)
Out[3]:
tensor([[ 0.2760, -0.9543, -1.0499,  0.7828],
        [ 1.3514, -1.1289,  0.5052, -0.0547]])

In [4]: x[[0,2]]
Out[4]:
tensor([[ 0.2760, -0.9543, -1.0499,  0.7828],
        [ 1.3514, -1.1289,  0.5052, -0.0547]])

I am puzzled about these two methods and look for some doc. But I failed. Can anyone can tell me are there some differences between them and what are these difference?

Output:
This looks like a remnant of old (slower) indexing.

See this pull request.

I also think you used to not be able to do binary logical indexing on tensors.

a = torch.randn((1,3,4,4))
dim = 2
indices = [0,1]

%timeit a.index_select(dim, torch.tensor(indices))

12.7 µs ± 1.28 µs per loop (mean ± std. dev. of 7 runs, 100000 loops each)

%timeit a[:,:,indices,:]

16.7 µs ± 640 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)