Instruction: Why am getting pylint import and no member errors when I didn't before?

Input: Hi I've been working on this code for months without any of the pylint errors that have suddenly appeared? How do I fix this?

Output:
You need pandas and torch installed in the same environment than the one you run pylint.
From the documentation at https://pylint.pycqa.org/en/latest/user_guide/messages/error/no-member.html:
If you are getting the dreaded no-member error, there is a possibility that either:

pylint found a bug in your code
You're launching pylint without the dependencies installed in its environment.
pylint would need to lint a C extension module and is refraining to do so.

Linting C extension modules is not supported out of the box, especially since pylint has no way to get an AST object out of the extension module.
But pylint actually has a mechanism which you might use in case you want to analyze C extensions. Pylint has a flag, called extension-pkg-allow-list (formerly extension-pkg-whitelist), through which you can tell it to import that module and to build an AST from that imported module:
 pylint --extension-pkg-allow-list=your_c_extension

Be aware though that using this flag means that extensions are loaded into the active Python interpreter and may run arbitrary code, which you may not want. This is the reason why we disable by default loading C extensions. In case you do not want the hassle of passing C extensions module with this flag all the time, you can enable unsafe-load-any-extension in your configuration file, which will build AST objects from all the C extensions that pylint encounters:
pylint --unsafe-load-any-extension=y

Alternatively, since pylint emits a separate error for attributes that cannot be found in C extensions, c-extension-no-member, you can disable this error for your project.


Instruction: Pip could not find a version that satisfies the requirement torch (from versions: none)

Input:
I am trying to install PyTorch on my Windows 11. I have Python 3.8.1, and pip 22.2.2. I have tried running the command py -m pip install torch, but it keeps returning the error:
ERROR: Could not find a version that satisfies the requirement torch (from versions: none)
ERROR: No matching distribution found for torch

I have gone through various StackOverflow posts and tried many different strategies. I even uninstalled and re-installed pip on my system using the get-pip.py file. Nothing seems to work.
Suggestions?

Output:
Just came across another post that mentioned how it needs to be the 64-bit version of Python to allow installation of PyTorch.
Just installed Python 3.9.13 with the 64-bit installer, and the installation worked. So, if anyone else is running into issues, would recommend first running python in command shell and checking whether it is 64 bit or 32 bit.


Instruction: Resizing inputs for torch model

Input:
I'm facing with this error properly and I could not see any exact solution or a solution formula for this error. My inputs are like (48x48) and that's not matching with the input shape of the resnet101. How can I edit my input to fit to the resnet101? You can see my code below, it probably helps you to understand my problem.
if __name__ == "__main__":
    vid = cv2.VideoCapture(0)
    emotions = []
    while vid.isOpened():
        image = cv2.imread("/home/berkay/Desktop/angry_man.jpg")
        _, frame = vid.read()

        # takes in a gray coloured filter of the frame
        gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

        # initializing the haarcascade face detector
        faces = face_cascade.detectMultiScale(frame)
        for (x,y,w,h) in faces:

            # takes the region of interest of the face only in gray
            roi_gray = gray[y:y+h, x:x+h]
            resized = cv2.resize(roi_gray, (48, 48))    # resizes to 48x48 sized image

            # predict the mood
            img = img2tensor(resized)
            prediction = predict(img)

In that point, I'm getting this error:
weight of size [64, 3, 7, 7], expected input[1, 1, 229, 229] to have 3 channels, but got 1 channels instead

How can I fix this? Thanks in advance

Output:
You can modify the input layer of resnet so that it would accept a single-channel tensors inputs using
In [1]: model = resnet101()
In [2]: model.conv1 = nn.Conv2d(1, 64, kernel_size=(2, 2))
In [3]: model(torch.rand(10, 1, 48, 48))
Out[3]:
tensor([[-0.5015,  0.6124,  0.1370,  ...,  1.2181, -0.4707,  0.3285],
        [-0.4776,  1.1027,  0.0161,  ...,  0.6363, -0.4733,  0.6218],
        [-0.3935,  0.8276, -0.0316,  ...,  0.6853, -0.4735,  0.6424],
        ...,
        [-0.2986,  1.1758,  0.0158,  ...,  0.7422, -0.4422,  0.4792],
        [-0.2668,  0.7884, -0.1205,  ...,  1.1445, -0.6249,  0.6697],
        [-0.2139,  1.0412,  0.2326,  ...,  0.8332, -0.8744,  0.4827]],
       grad_fn=<AddmmBackward0>)

(you will probably need to modify the kernel size accordingly too)


Instruction: PyTorch | loss.backward() -> Missing XLA configuration

Input:
The loss is calculated from the target model created using pytorch (not TensorFlow) and when propagating, I run the code below and had trouble with the following error message.
loss.backward()

(Forward propagation can be calculated without problems.)
terminate called after throwing an instance of 'std::runtime_error'
  what(): tensorflow/compiler/xla/xla_client/computation_client.cc:280 : Missing XLA configuration
Aborted

-pytorch(1.12.0+cu102)

torchvision(0.13.0+cu102) <- target model contains pre-trained CNN model which can be installed from torchvision.models
google-compute-engine
GPU (NVIDIA Tesla T4 x 1, 11.6) <- The code worked in the environment where GPU (11.2) was installed, but it does not work in the current environment. / In the current environment, the same error occurs even if the GPU is not used and the CPU is used.
TPU is not installed (I don't want to use TPU, but GPU)

The code is working locally and was also working on other GPU environments as mentioned above. It stopped working when the environment was updated.
Please help me···

Output:
I solved this problem with the command.
$ pip uninstall torch_xla

This error seemed to be caused by pytorch-ignite and torch_xla.


Instruction: Applying pre-trained BERT model to make predictions

Input:
I have recently been given a BERT model that has been pre-trained with a mental health dataset that I have. Now all I have to do is apply the model to a larger dataset to test its performance. I am absolutely new to machine learning and am stuck in this step. I am using PyTorch and would like to continue using it. Here is what I have tried so far:

    df = pd.read_csv('C:/Users/X/Downloads/number_users_09_05.csv',  encoding='unicode-
    escape')

    REPLACE_BY_SPACE_RE = re.compile('[/(){}\[\]\|@,;*]')
    BAD_SYMBOLS_RE = re.compile('[^0-9a-z #+_]')
    STOPWORDS = set(stopwords.words('english'))

    # function to clean text
    def clean_text(text):
        text = text.lower()
        text = REPLACE_BY_SPACE_RE.sub(' ', text)
        text = BAD_SYMBOLS_RE.sub(' ', text)
        text = ' '.join(word for word in text.split() if word not in STOPWORDS)
        return text

    df['cleaned_text'] = df['post'].apply(clean_text)

    One-Hot Encoding

    df['Categories'] = df['Categories'].astype(str)

    from sklearn.preprocessing import MultiLabelBinarizer
    df['Categories'] = df['Categories'].str.split(",").tolist()
    multilabel_binarizer = MultiLabelBinarizer()
    y = multilabel_binarizer.fit_transform(df['Categories'])
    print(multilabel_binarizer.classes_)

    for idx, Categories in enumerate(multilabel_binarizer.classes_):
        df[Categories] = y[:,idx]
    X = df['cleaned_text'].values

    from sklearn.model_selection import train_test_split
    x_tr,x_val,y_tr,y_val=train_test_split(X, y, test_size=0.2, random_state=0,shuffle=True)

    input_dir = 'C:/Users/X/Documents/ppd_bert_model/'

    from transformers import AutoModelForMaskedLM
    model = AutoModelForMaskedLM.from_pretrained(input_dir)

    from transformers import AutoTokenizer
    tokenizer = AutoTokenizer.from_pretrained(input_dir)

    model.load_weights('C:/Users/X/Documents/ppd_bert_model/')

At this step I get the following error:
AttributeError                            Traceback (most recent call last)
Input In [30], in <cell line: 1>()
----> 1 model.load_weights('C:/Users/X/Documents/ppd_bert_model/')
      2 pred_prob = model.predict(x_val_seq)

File ~\Anaconda3\lib\site-packages\torch\nn\modules\module.py:1207, in Module.__getattr__(self, name)
   1205     if name in modules:
   1206         return modules[name]
-> 1207 raise AttributeError("'{}' object has no attribute '{}'".format(
   1208     type(self).__name__, name))

AttributeError: 'BertForMaskedLM' object has no attribute 'load_weights'

Then after this I tried:
    pred_prob = model.predict(df['cleaned_text'])

And got this error:
AttributeError                            Traceback (most recent call last)
Input In [28], in <cell line: 1>()
----> 1 pred_prob = model.predict(df['cleaned_text'])

File ~\Anaconda3\lib\site-packages\torch\nn\modules\module.py:1207, in Module.__getattr__(self, name)
   1205     if name in modules:
   1206         return modules[name]
-> 1207 raise AttributeError("'{}' object has no attribute '{}'".format(
   1208     type(self).__name__, name))

AttributeError: 'BertForMaskedLM' object has no attribute 'predict'

I have the pretrained model saved in a folder at 'C:/Users/X/Documents/ppd_bert_model/', and the model has three files: config.json, pytorch_model.bin, and vocab.txt. If anyone has any advice on how I should be calling and using the model to make predictions and test performance, please do let me know.
Thank you in advance!

Output:
From the HuggingFace documentation here, the AutoModelForMaskedLM class uses the from_pretrained() method to instantiate one of the model classes of the library (with a masked language modeling head) from a pretrained model. This is done either by using the model_type property in the config object (either passed as an argument, or loaded from pretained_model_name_or_path (in your case "C:/Users/X/Documents/ppd_bert_model/"), or if the model_type property is missing, it will fall back to pattern matching on pretrained_model_name_or_path.
Based on your first error AttributeError: 'BertForMaskedLM' object has no attribute 'load_weights', the BertForMaskedLM is being automatically loaded and an associated class instantiated by the from_pretrained() method call. If you follow the documentation I've hyperlinked, and look at the BertForMaskedLM class, there is no method load_weights(), hence the first error you encountered with your method call. You'll also see in the documentation that there is no predict() method, hence the second error you encountered. At the bottom of that subsection you'll find an example code snippet that shows you how to use the BertForMaskedLM model for masked token prediction. I'll reproduce the snippet here for your convenience and future reference, should their documentation change or the link break:
>>> from transformers import BertTokenizer, BertForMaskedLM
>>> import torch

>>> tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
>>> model = BertForMaskedLM.from_pretrained("bert-base-uncased")

>>> inputs = tokenizer("The capital of France is [MASK].", return_tensors="pt")

>>> with torch.no_grad():
...    logits = model(**inputs).logits

>>> # retrieve index of [MASK]
>>> mask_token_index = (inputs.input_ids == tokenizer.mask_token_id)[0].nonzero(as_tuple=True)[0]

>>> predicted_token_id = logits[0, mask_token_index].argmax(axis=-1)

>>> tokenizer.decode(predicted_token_id)
'paris'

Hopefully this helps get you on the right track towards making predictions and testing performance.


Instruction: Basic IPU example crashes with ValueError: Expected a parent

Input:
I wanted to test the free IPU runtime on Paperspace, as such I made a free account and selected the HuggingFace + IPU notebook.
Afterwards I created the following very simple notebook with Pytorch Lightning to perform classification on MNIST (the simplest possible example):
!python3 -m pip install torchvision==0.11.1
!python3 -m pip install pytorch_lightning

import torch
from torch.nn import functional as F

import pytorch_lightning as pl
from torch.utils.data import DataLoader
import torchvision
import poptorch

class LitClassifier(pl.LightningModule):
    def __init__(self, hidden_dim: int = 128, learning_rate: float = 0.0001):
        super().__init__()
        self.save_hyperparameters()

        self.l1 = torch.nn.Linear(28 * 28, self.hparams.hidden_dim)
        self.l2 = torch.nn.Linear(self.hparams.hidden_dim, 10)

    def forward(self, x):
        x = x.view(x.size(0), -1)
        x = torch.relu(self.l1(x))
        x = torch.relu(self.l2(x))
        return x

    def training_step(self, batch, batch_idx):
        x, y = batch
        y_hat = self(x)
        loss = F.cross_entropy(y_hat, y)
        return loss

    def validation_step(self, batch, batch_idx):
        x, y = batch
        probs = self(x)
        # we currently return the accuracy as the validation_step/test_step is run on the IPU devices.
        # Outputs from the step functions are sent to the host device, where we calculate the metrics in
        # validation_epoch_end and test_epoch_end for the test_step.
        acc = self.accuracy(probs, y)
        return acc

    def test_step(self, batch, batch_idx):
        x, y = batch
        logits = self(x)
        acc = self.accuracy(logits, y)
        return acc

    def accuracy(self, logits, y):
        # currently IPU poptorch doesn't implicit convert bools to tensor
        # hence we use an explicit calculation for accuracy here. Once fixed in poptorch
        # we can use the accuracy metric.
        acc = torch.sum(torch.eq(torch.argmax(logits, -1), y).to(torch.float32)) / len(y)
        return acc

    def validation_epoch_end(self, outputs) -> None:
        # since the training step/validation step and test step are run on the IPU device
        # we must log the average loss outside the step functions.
        self.log("val_acc", torch.stack(outputs).mean(), prog_bar=True)

    def test_epoch_end(self, outputs) -> None:
        self.log("test_acc", torch.stack(outputs).mean())

    def configure_optimizers(self):
        return torch.optim.Adam(self.parameters(), lr=self.hparams.learning_rate)

training_batch_size = 10

dm = DataLoader(
     torchvision.datasets.MNIST('mnist_data/',
                                train=True,
                                download=True,
                                transform=torchvision.transforms.Compose([
                                    torchvision.transforms.ToTensor(),
                                    torchvision.transforms.Normalize(
                                        (0.1307, ), (0.3081, ))
                                ])),
     batch_size=training_batch_size,
     shuffle=True)

model = LitClassifier()

print(model)
trainer = pl.Trainer(max_epochs=2, accelerator="ipu", devices="auto")

trainer.fit(model, datamodule=dm)

The code crashes with what looks like an internal error of the library with:
LitClassifier(
  (l1): Linear(in_features=784, out_features=128, bias=True)
  (l2): Linear(in_features=128, out_features=10, bias=True)
)
GPU available: False, used: False
TPU available: False, using: 0 TPU cores
IPU available: True, using: 4 IPUs
HPU available: False, using: 0 HPUs
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-24-329fa233a013> in <module>
     83 trainer = pl.Trainer(max_epochs=2, accelerator="ipu", devices="auto")
     84
---> 85 trainer.fit(model, datamodule=dm)

/usr/local/lib/python3.8/dist-packages/pytorch_lightning/trainer/trainer.py in fit(self, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path)
    694         """
    695         self.strategy.model = model
--> 696         self._call_and_handle_interrupt(
    697             self._fit_impl, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path
    698         )

/usr/local/lib/python3.8/dist-packages/pytorch_lightning/trainer/trainer.py in _call_and_handle_interrupt(self, trainer_fn, *args, **kwargs)
    648                 return self.strategy.launcher.launch(trainer_fn, *args, trainer=self, **kwargs)
    649             else:
--> 650                 return trainer_fn(*args, **kwargs)
    651         # TODO(awaelchli): Unify both exceptions below, where `KeyboardError` doesn't re-raise
    652         except KeyboardInterrupt as exception:

/usr/local/lib/python3.8/dist-packages/pytorch_lightning/trainer/trainer.py in _fit_impl(self, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path)
    733             ckpt_path, model_provided=True, model_connected=self.lightning_module is not None
    734         )
--> 735         results = self._run(model, ckpt_path=self.ckpt_path)
    736
    737         assert self.state.stopped

/usr/local/lib/python3.8/dist-packages/pytorch_lightning/trainer/trainer.py in _run(self, model, ckpt_path)
   1089         self._callback_connector._attach_model_logging_functions()
   1090
-> 1091         verify_loop_configurations(self)
   1092
   1093         # hook

/usr/local/lib/python3.8/dist-packages/pytorch_lightning/trainer/configuration_validator.py in verify_loop_configurations(trainer)
     57     _check_on_pretrain_routine(model)
     58     # TODO: Delete CheckpointHooks off LightningDataModule in v1.8
---> 59     _check_datamodule_checkpoint_hooks(trainer)
     60     _check_setup_method(trainer)
     61

/usr/local/lib/python3.8/dist-packages/pytorch_lightning/trainer/configuration_validator.py in _check_datamodule_checkpoint_hooks(trainer)
    291
    292 def _check_datamodule_checkpoint_hooks(trainer: "pl.Trainer") -> None:
--> 293     if is_overridden(method_name="on_save_checkpoint", instance=trainer.datamodule):
    294         rank_zero_deprecation(
    295             "`LightningDataModule.on_save_checkpoint` was deprecated in"

/usr/local/lib/python3.8/dist-packages/pytorch_lightning/utilities/model_helpers.py in is_overridden(method_name, instance, parent)
     32             parent = pl.Callback
     33         if parent is None:
---> 34             raise ValueError("Expected a parent")
     35
     36     instance_attr = getattr(instance, method_name, None)

ValueError: Expected a parent

Is this a problem of incompatibility of the versions of the libraries? I tried searching on Google this error but found only this question: pytorch - Model_heplers.py in is_overridden > raise ValueError(“Expected a parent”) but I do not think that it is my case because I just use built-in Dataloader and inherited from pl.LightningModule for my network. The example in paperspace docs: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/example.html works but it does not use Pytorch Lightning.
Is there a way to make this service behave correctly with Lightning?

Output:
To make Pytorch Lightning correctly handle the dataloader in this case it must be put inside the LitClassifier Class.
!python3 -m pip install torchvision==0.11.1
!python3 -m pip install pytorch_lightning

import torch
from torch.nn import functional as F

import pytorch_lightning as pl
from torch.utils.data import DataLoader
import torchvision
import poptorch

class LitClassifier(pl.LightningModule):
    def __init__(self, hidden_dim: int = 128, learning_rate: float = 0.0001):
        super().__init__()
        self.save_hyperparameters()

        self.l1 = torch.nn.Linear(28 * 28, self.hparams.hidden_dim)
        self.l2 = torch.nn.Linear(self.hparams.hidden_dim, 10)

    def forward(self, x):
        x = x.view(x.size(0), -1)
        x = torch.relu(self.l1(x))
        x = torch.relu(self.l2(x))
        return x

    def training_step(self, batch, batch_idx):
        x, y = batch
        y_hat = self(x)
        loss = F.cross_entropy(y_hat, y)
        return loss

    def validation_step(self, batch, batch_idx):
        x, y = batch
        probs = self(x)
        # we currently return the accuracy as the validation_step/test_step is run on the IPU devices.
        # Outputs from the step functions are sent to the host device, where we calculate the metrics in
        # validation_epoch_end and test_epoch_end for the test_step.
        acc = self.accuracy(probs, y)
        return acc

    def test_step(self, batch, batch_idx):
        x, y = batch
        logits = self(x)
        acc = self.accuracy(logits, y)
        return acc

    def accuracy(self, logits, y):
        # currently IPU poptorch doesn't implicit convert bools to tensor
        # hence we use an explicit calculation for accuracy here. Once fixed in poptorch
        # we can use the accuracy metric.
        acc = torch.sum(torch.eq(torch.argmax(logits, -1), y).to(torch.float32)) / len(y)
        return acc

    def validation_epoch_end(self, outputs) -> None:
        # since the training step/validation step and test step are run on the IPU device
        # we must log the average loss outside the step functions.
        self.log("val_acc", torch.stack(outputs).mean(), prog_bar=True)

    def test_epoch_end(self, outputs) -> None:
        self.log("test_acc", torch.stack(outputs).mean())

    def configure_optimizers(self):
        return torch.optim.Adam(self.parameters(), lr=self.hparams.learning_rate)

    def train_dataloader(self):
        training_batch_size = 100
        return DataLoader(
     torchvision.datasets.MNIST('mnist_data/',
                                train=True,
                                download=True,
                                transform=torchvision.transforms.Compose([
                                    torchvision.transforms.ToTensor(),
                                    torchvision.transforms.Normalize(
                                        (0.1307, ), (0.3081, ))
                                ])),
     batch_size=training_batch_size,
     num_workers=240,
     shuffle=True)

    def test_dataloader(self):
        val_batch_size = 100
        return DataLoader(
     torchvision.datasets.MNIST('mnist_data/',
                                train=False,
                                download=True,
                                transform=torchvision.transforms.Compose([
                                    torchvision.transforms.ToTensor(),
                                    torchvision.transforms.Normalize(
                                        (0.1307, ), (0.3081, ))
                                ])),
     batch_size=val_batch_size,
     num_workers=240,
     shuffle=False)

model = LitClassifier()

print(model)
trainer = pl.Trainer(max_epochs=2, accelerator="ipu", devices="auto")

trainer.fit(model)


Instruction: Convert PyTorch AutoTokenizer to TensorFlow TextVectorization

Input:
I have a PyTorch encoder loaded on my PC with transformers.
I saved it in JSON with tokenizer.save_pretrained(...) and now I need to load it on another PC with TensorFlow TextVectorization as I don't have access to the transformers library.
How can I convert ? I read about the tf.keras.preprocessing.text.tokenizer_from_json but it does not work.
In PyTorch JSON I have :
{
  "version": "1.0",
  "truncation": null,
  "padding": null,
  "added_tokens": [...],
  "normalizer": {...},
  "pre_tokenizer": {...},
  "post_processor": {...},
  "decoder": {...},
  "model": {...}
}

and TensorFlow is expecting, with TextVectorizer :
    def __init__(
        self,
        max_tokens=None,
        standardize="lower_and_strip_punctuation",
        split="whitespace",
        ngrams=None,
        output_mode="int",
        output_sequence_length=None,
        pad_to_max_tokens=False,
        vocabulary=None,
        idf_weights=None,
        sparse=False,
        ragged=False,
        **kwargs,
    ):

or with the tokenizer_from_json these kind of fields :
    config = tokenizer_config.get("config")

    word_counts = json.loads(config.pop("word_counts"))
    word_docs = json.loads(config.pop("word_docs"))
    index_docs = json.loads(config.pop("index_docs"))
    # Integer indexing gets converted to strings with json.dumps()
    index_docs = {int(k): v for k, v in index_docs.items()}
    index_word = json.loads(config.pop("index_word"))
    index_word = {int(k): v for k, v in index_word.items()}
    word_index = json.loads(config.pop("word_index"))

    tokenizer = Tokenizer(**config)

Output:
Simply "tf.keras.preprocessing.text.tokenizer_from_json.()" but you may need to correct format in JSON.
Sample: The sample they using " I love cats " -> " Sticky "
import tensorflow as tf

text = "I love cats"
tokenizer = tf.keras.preprocessing.text.Tokenizer(num_words=10000, oov_token='<oov>')
tokenizer.fit_on_texts([text])

# input
vocab = [ "a", "b", "c", "d", "e", "f", "g", "h", "I", "j", "k", "l", "m", "n", "o", "p", "q", "r", "s", "t", "u", "v", "w", "x", "y", "z", "_" ]
data = tf.constant([["_", "_", "_", "I"], ["l", "o", "v", "e"], ["c", "a", "t", "s"]])

layer = tf.keras.layers.StringLookup(vocabulary=vocab)
sequences_mapping_string = layer(data)
sequences_mapping_string = tf.constant( sequences_mapping_string, shape=(1,12) )
print( 'result: ' + str( sequences_mapping_string ) )

print( 'tokenizer.to_json(): ' + str( tokenizer.to_json() ) )

new_tokenizer = tf.keras.preprocessing.text.tokenizer_from_json(tokenizer.to_json())
print( 'new_tokenizer.to_json(): ' + str( new_tokenizer.to_json() ) )

Output:
result: tf.Tensor([[27 27 27  9 12 15 22  5  3  1 20 19]], shape=(1, 12), dtype=int64)
tokenizer.to_json(): {"class_name": "Tokenizer", "config": {"num_words": 10000, "filters": "!\"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\t\n", "lower": true, "split": " ", "char_level": false, "oov_token": "<oov>", "document_count": 1, "word_counts": "{\"i\": 1, \"love\": 1, \"cats\": 1}", "word_docs": "{\"cats\": 1, \"love\": 1, \"i\": 1}", "index_docs": "{\"4\": 1, \"3\": 1, \"2\": 1}", "index_word": "{\"1\": \"<oov>\", \"2\": \"i\", \"3\": \"love\", \"4\": \"cats\"}", "word_index": "{\"<oov>\": 1, \"i\": 2, \"love\": 3, \"cats\": 4}"}}
new_tokenizer.to_json(): {"class_name": "Tokenizer", "config": {"num_words": 10000, "filters": "!\"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\t\n", "lower": true, "split": " ", "char_level": false, "oov_token": "<oov>", "document_count": 1, "word_counts": "{\"i\": 1, \"love\": 1, \"cats\": 1}", "word_docs": "{\"cats\": 1, \"love\": 1, \"i\": 1}", "index_docs": "{\"4\": 1, \"3\": 1, \"2\": 1}", "index_word": "{\"1\": \"<oov>\", \"2\": \"i\", \"3\": \"love\", \"4\": \"cats\"}", "word_index": "{\"<oov>\": 1, \"i\": 2, \"love\": 3, \"cats\": 4}"}}


Instruction: What is the proper input dimension for torch.nn.BCEWithLogitsLoss?

Input: I am training a binary classifier and using torch.nn.BCEWithLogitsLoss as the loss function. I am confused about the proper input dimension for the loss function. Should it be [n, 1] or [n, 2]? In the case of [n, 1], where n is the number of samples, the values for the targets would just be 0 or 1, which represents the class that the sample belongs to. In the case of [n, 2], the targets would be torch.nn.functional.one_hot(targets, num_classes=2).float(). Which one is the proper dimension and what would be the corresponding logits and/or last layer of my network?

Output:
The documentation tells you all you need to know.
For a prediction tensor x of shape (num_samples, num_classes), the target tensor y should of the exact same shape.
Case 1
If your number of classes is two, and they are mutually exclusive, ax of shape (num_samples,) should be contrasted with a y of shape (num_samples,) taking values 0 | 1 (but typecast to float).
Case 2
If your number of classes is two, but they are not mutually exclusive (multi-class classification), x and y should be of shape (num_samples, 2) with y still taking float values from 0 | 1.
In either case, you need a final layer that projects your network's dimensionality to your number of classes (whatever that means in your context). So in case 1, something like Linear(model_dim, 1), whereas in case 2 Linear(model_dim, 2).
Remember not to apply any activation function to your network's output, as this is incorporated in BCEWithLogitsLoss already.
If your target labels are not discrete, adjust accordingly (values from [0..1] rather than 0 | 1).


Instruction: CIFAR-10 CNN using PyTorch

Input:
I'm working on CIFAR-10 dataset using PyTorch and observed some categories have low accuracy around  40-50%. Why do some categories of image were more difficult to work with?
enter image description here

Output: The reason for low accuracy for some classes often has to do with significant similarity with other classes.  For example, cat and dog may be confused for each other fairly often by the model.  Your confusion matrix shows what types of mistakes your model is making.  To some extent, it is confirming this intuition that cats and dogs are often confused (as wells as for other animals).  I would suspect that the similarity in the "domestic" background of these domesticated animals is also contributing to the confusion.


Instruction: What does num_labels actually do?

Input:
When training a BERT based model one can set num_labels
AutoConfig.from_pretrained(BERT_MODEL_NAME, num_labels=num_labels)
So for example if we want to have a prediction of 3 values we may use num_labels=3.
My question is what does it do internally? Is it just connecting a nn.Linear to the last embedding layer?
Thanks

Output:
I suppose if there is a num label then the model is used for classification then simply you can go to the documentation of BERT on hugging face then search for the classification class and take a look into the code, then you will find the following:
https://github.com/huggingface/transformers/blob/bd469c40659ce76c81f69c7726759d249b4aef49/src/transformers/models/bert/modeling_bert.py#L1572
        if labels is not None:
        if self.config.problem_type is None:
            if self.num_labels == 1:
                self.config.problem_type = "regression"
            elif self.num_labels > 1 and (labels.dtype == torch.long or labels.dtype == torch.int):
                self.config.problem_type = "single_label_classification"
            else:
                self.config.problem_type = "multi_label_classification"

        if self.config.problem_type == "regression":
            loss_fct = MSELoss()
            if self.num_labels == 1:
                loss = loss_fct(logits.squeeze(), labels.squeeze())
            else:
                loss = loss_fct(logits, labels)
        elif self.config.problem_type == "single_label_classification":
            loss_fct = CrossEntropyLoss()
            loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
        elif self.config.problem_type == "multi_label_classification":
            loss_fct = BCEWithLogitsLoss()
            loss = loss_fct(logits, labels)

so the number of labels as we see affects using the loss function
hope this answers your question


Instruction: Different cross entropy results from NumPy and PyTorch

Input:
My prediction is y_hat = [ 0.57,0.05,0.14,0.10,0.14] and target is
target =[ 1, 0, 0, 0, 0 ].
I need to calculate Cross Entropy loss by NumPy and Pytorch loss function.
Using NumPy my formula is -np.sum(target*np.log(y_hat)), and I got 0.5621189181535413
However, using Pytorch:
loss = nn.CrossEntropyLoss()
output = torch.FloatTensor([0.57,0.05,0.14,0.10,0.14])
label = torch.FloatTensor([1,0,0,0,0])
loss_value = loss(output, label)
print(loss_value)

Gives tensor(1.2586), which is different.

Output:
You need to apply the softmax function to your y_hat vector before computing cross-entropy loss. For example, you can use scipy.special.softmax().
>>> from scipy.special import softmax
>>> import numpy as np
>>> y_hat = [0.57, 0.05, 0.14, 0.10, 0.14]
>>> target =[1, 0, 0, 0, 0]
>>> y_hat = softmax(y_hat)
>>> -np.sum(target * np.log(y_hat))
1.2586146726011722

Which agrees with the result from Pytorch.


Instruction: How can I add reshape layer in nn.Sequential?

Input:
So I'm implementing Generator of a GAN and I need the architecture as shown as below:

The problem is when I try to reshape the output of Linear layer after BatchNorm and ReLU (in fig. Dense as they have used Tensorflow) it is throwing error as :TypeError: reshape(): argument 'input' (position 1) must be Tensor, not int
I understand the error but I can't find its solution.
Is there any other way to reshape within nn.Sequential instead of calling torch explicitly?
class Generator(nn.Module):

    def __init__(self, z_dim=100, im_chan=1, hidden_dim=64, rdim=9216):

        super(Generator, self).__init__()
        self.z_dim = z_dim
        self.gen = nn.Sequential(
             nn.Linear(z_dim, rdim),
             nn.BatchNorm2d(rdim,momentum=0.9),
             nn.ReLU(inplace=True),
 ---->       torch.reshape(rdim, (6,6,256)),
             self.make_gen_block(rdim, hidden_dim*2),
             self.make_gen_block(hidden_dim*2,hidden_dim),
             self.make_gen_block(hidden_dim,im_chan,final_layer=True),
        )
   def make_gen_block(self, input_channels, output_channels, kernel_size=1, stride=2, final_layer=False):

        if not final_layer:
           return nn.Sequential(
              nn.ConvTranspose2d(input_channels, output_channels, kernel_size, stride),
              nn.BatchNorm2d(output_channels),
              nn.ReLU(inplace=True)
        )
        else:
              return nn.Sequential(
              nn.ConvTranspose2d(input_channels, output_channels, kernel_size, stride),
              nn.Tanh()
        )

   def unsqueeze_noise(self, noise):
       return noise.view(len(noise), self.zdim, 1, 1)

   def forward(self, noise):
       x = self.unsqueeze_noise(noise)
       return self.gen(x)

def get_noise(n_samples, z_dim, device='cpu'):
    return torch.randn(n_samples, z_dim, device=device)

#Testing the Gen arch
gen = Generator()
num_test = 100

#test the hidden block
test_hidden_noise = get_noise(num_test, gen.z_dim)
test_hidden_block = gen.make_gen_block(6, 6, kernel_size=1,stride=2)
test_uns_noise = gen.unsqueeze_noise(test_hidden_noise)
hidden_output = test_hidden_block(test_uns_noise)

Output:
In nn.Sequential, torch.nn.Unflatten() can help you achieve reshape operation.

For nn.Linear, its input shape is (N, *, H_{in}) and output shape is (H, *, H_{out}). Note that the feature dimension is last. So unsqueeze_noise() is not useful here.

Based on the network structure, the arguments passed to make_gen_block are wrong.

I have checked the following code:
import torch
from torch import nn
class Generator(nn.Module):

   def __init__(self, z_dim=100, im_chan=1, hidden_dim=64, rdim=9216):

        super(Generator, self).__init__()
        self.z_dim = z_dim
        self.gen = nn.Sequential(
             nn.Linear(z_dim, rdim),
             nn.BatchNorm1d(rdim,momentum=0.9), # use BN1d
             nn.ReLU(inplace=True),
             nn.Unflatten(1, (256,6,6)),
             self.make_gen_block(256, hidden_dim*2,kernel_size=2), # note arguments
             self.make_gen_block(hidden_dim*2,hidden_dim,kernel_size=2), # note kernel_size
             self.make_gen_block(hidden_dim,im_chan,kernel_size=2,final_layer=True), # note kernel_size
        )
   def make_gen_block(self, input_channels, output_channels, kernel_size=1, stride=2, final_layer=False):

        if not final_layer:
           return nn.Sequential(
              nn.ConvTranspose2d(input_channels, output_channels, kernel_size, stride),
              nn.BatchNorm2d(output_channels),
              nn.ReLU(inplace=True)
        )
        else:
              return nn.Sequential(
              nn.ConvTranspose2d(input_channels, output_channels, kernel_size, stride),
              nn.Tanh()
        )

   def forward(self, x):
       return self.gen(x)

def get_noise(n_samples, z_dim, device='cpu'):
    return torch.randn(n_samples, z_dim, device=device)

gen = Generator()
num_test = 100
input_noise = get_noise(num_test, gen.z_dim)
output = gen(input_noise)
assert output.shape == (num_test, 1, 48, 48)


Instruction: Torch Conv 1-D Layer outputs the same value repeated

Input:
I have a torch tensors of shape (768, 8, 22), procesed by a Conv Layer 1-D Layer. The data is already standarized. The tensor is processed by the next layer:
nn.Conv1d(input_chanels = 8, output_chanels = 1, kernel_size = 3)
The output of the layer has the right dimensions, but the output matrix has the same value repeated over and over again.
output = tensor(

         [[[-0.1856, -0.1856, -0.1856, -0.1856, -0.1856, -0.1856, -0.1856,
          -0.1856, -0.1856, -0.1856, -0.1856, -0.1856, -0.1856, -0.1856,
          -0.1856, -0.1856, -0.1856, -0.1856, -0.1856, -0.1856]],
          ...
          [[-0.1856, -0.1856, -0.1856, -0.1856, -0.1856, -0.1856, -0.1856,
          -0.1856, -0.1856, -0.1856, -0.1856, -0.1856, -0.1856, -0.1856,
          -0.1856, -0.1856, -0.1856, -0.1856, -0.1856, -0.1856]]]

Output: I've solved this issue, it's a weird situation of torch. I just remove de Conv1dfrom the init from my network class and I defined in the forward method. By doing so, the output is no longer the same for every step.


Instruction: How to test pytorch GPU code on a CPU machine

Input:
I want to verify on CPU machine that I've moved all tensors to CUDA device correctly. Is there a way to create a mock device that is still computed on CPU but marked to be incompatible with tensors I forget to move to this device?
Motivation: this would speed up my development cycle as I don't need to deploy code and run it on the GPU server only to find out I forgot to specify device on some tensor / model.

Output:
Maybe the meta device is what you need.
>>> a = torch.ones(2) # defaults to cpu
>>> b = torch.ones(2, device='meta')
>>> c = a + b
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, meta and cpu!

Note that original intention for meta device is to compute shape and dtype of output without actually doing full computation. Tensors on meta device have no data in them, so I am not sure if there are some aspects of the test where meta device behaves different from a "real" device.
Interestingly I couldn't find much documentation about this meta device at all. I found it mentioned here https://github.com/pytorch/pytorch/issues/61654#issuecomment-879989145


Instruction: How to stop logging locally but only save to wandb's servers and have wandb work using soft links?

Input:
I am having a weird issue where I change the location of all my code & data to a different location with more disk space, then I soft link my projects & data to those locations with more space. I assume there must be some file handle issue because wandb's logger is throwing me issues. So my questions:

how do I have wandb only log  online and not locally? (e.g. stop trying to log anything to ./wandb[or any secret place it might be logging to]  since it's creating issues). Note my code was running fine after I  stopped logging to wandb so I assume that was the issue. note that the dir=None is the default to wandb's param.
how do I resolve this issue entirely so that it works seemlessly with all my projects softlinked somewhere else?

More details on the error
Traceback (most recent call last):
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/logging/__init__.py", line 1087, in emit
    self.flush()
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/logging/__init__.py", line 1067, in flush
    self.stream.flush()
OSError: [Errno 116] Stale file handle
Call stack:
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/threading.py", line 930, in _bootstrap
    self._bootstrap_inner()
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/threading.py", line 973, in _bootstrap_inner
    self.run()
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/wandb/vendor/watchdog/observers/api.py", line 199, in run
    self.dispatch_events(self.event_queue, self.timeout)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/wandb/vendor/watchdog/observers/api.py", line 368, in dispatch_events
    handler.dispatch(event)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/wandb/vendor/watchdog/events.py", line 454, in dispatch
    _method_map[event_type](event)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/wandb/filesync/dir_watcher.py", line 275, in _on_file_created
    logger.info("file/dir created: %s", event.src_path)
Message: 'file/dir created: %s'
Arguments: ('/shared/rsaas/miranda9/diversity-for-predictive-success-of-meta-learning/wandb/run-20221023_170722-1tfzh49r/files/output.log',)
--- Logging error ---
Traceback (most recent call last):
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/logging/__init__.py", line 1087, in emit
    self.flush()
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/logging/__init__.py", line 1067, in flush
    self.stream.flush()
OSError: [Errno 116] Stale file handle
Call stack:
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/threading.py", line 930, in _bootstrap
    self._bootstrap_inner()
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/threading.py", line 973, in _bootstrap_inner
    self.run()
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/wandb/sdk/internal/internal_util.py", line 50, in run
    self._run()
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/wandb/sdk/internal/internal_util.py", line 101, in _run
    self._process(record)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/wandb/sdk/internal/internal.py", line 263, in _process
    self._hm.handle(record)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/wandb/sdk/internal/handler.py", line 130, in handle
    handler(record)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/wandb/sdk/internal/handler.py", line 138, in handle_request
    logger.debug(f"handle_request: {request_type}")
Message: 'handle_request: stop_status'
Arguments: ()
N/A% (0 of 100000) |      | Elapsed Time: 0:00:00 | ETA:  --:--:-- |   0.0 s/it

Traceback (most recent call last):
  File "/home/miranda9/diversity-for-predictive-success-of-meta-learning/div_src/diversity_src/experiment_mains/main_dist_maml_l2l.py", line 1814, in <module>
    main()
  File "/home/miranda9/diversity-for-predictive-success-of-meta-learning/div_src/diversity_src/experiment_mains/main_dist_maml_l2l.py", line 1747, in main
    train(args=args)
  File "/home/miranda9/diversity-for-predictive-success-of-meta-learning/div_src/diversity_src/experiment_mains/main_dist_maml_l2l.py", line 1794, in train
    meta_train_iterations_ala_l2l(args, args.agent, args.opt, args.scheduler)
  File "/home/miranda9/ultimate-utils/ultimate-utils-proj-src/uutils/torch_uu/training/meta_training.py", line 167, in meta_train_iterations_ala_l2l
    log_zeroth_step(args, meta_learner)
  File "/home/miranda9/ultimate-utils/ultimate-utils-proj-src/uutils/logging_uu/wandb_logging/meta_learning.py", line 92, in log_zeroth_step
    log_train_val_stats(args, args.it, step_name, train_loss, train_acc, training=True)
  File "/home/miranda9/ultimate-utils/ultimate-utils-proj-src/uutils/logging_uu/wandb_logging/supervised_learning.py", line 55, in log_train_val_stats
    _log_train_val_stats(args=args,
  File "/home/miranda9/ultimate-utils/ultimate-utils-proj-src/uutils/logging_uu/wandb_logging/supervised_learning.py", line 116, in _log_train_val_stats
    args.logger.log('\n')
  File "/home/miranda9/ultimate-utils/ultimate-utils-proj-src/uutils/logger.py", line 89, in log
    print(msg, flush=flush)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/wandb/sdk/lib/redirect.py", line 640, in write
    self._old_write(data)
OSError: [Errno 116] Stale file handle
wandb: Waiting for W&B process to finish... (failed 1). Press Control-C to abort syncing.
wandb: Synced vit_mi Adam_rfs_cifarfs Adam_cosine_scheduler_rfs_cifarfs 0.001: args.jobid=101161: https://wandb.ai/brando/entire-diversity-spectrum/runs/1tfzh49r
wandb: Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
wandb: Find logs at: ./wandb/run-20221023_170722-1tfzh49r/logs
--- Logging error ---
Traceback (most recent call last):
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/wandb/sdk/interface/router_sock.py", line 27, in _read_message
    resp = self._sock_client.read_server_response(timeout=1)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/wandb/sdk/lib/sock_client.py", line 283, in read_server_response
    data = self._read_packet_bytes(timeout=timeout)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/wandb/sdk/lib/sock_client.py", line 269, in _read_packet_bytes
    raise SockClientClosedError()
wandb.sdk.lib.sock_client.SockClientClosedError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/wandb/sdk/interface/router.py", line 70, in message_loop
    msg = self._read_message()
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/wandb/sdk/interface/router_sock.py", line 29, in _read_message
    raise MessageRouterClosedError
wandb.sdk.interface.router.MessageRouterClosedError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/logging/__init__.py", line 1087, in emit
    self.flush()
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/logging/__init__.py", line 1067, in flush
    self.stream.flush()
OSError: [Errno 116] Stale file handle
Call stack:
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/threading.py", line 930, in _bootstrap
    self._bootstrap_inner()
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/threading.py", line 973, in _bootstrap_inner
    self.run()
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/threading.py", line 910, in run
    self._target(*self._args, **self._kwargs)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/wandb/sdk/interface/router.py", line 77, in message_loop
    logger.warning("message_loop has been closed")
Message: 'message_loop has been closed'
Arguments: ()
/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/tempfile.py:817: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/srv/condor/execute/dir_27749/tmpmvf78q6owandb'>
  _warnings.warn(warn_message, ResourceWarning)
/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/tempfile.py:817: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/srv/condor/execute/dir_27749/tmpt5etqpw_wandb-artifacts'>
  _warnings.warn(warn_message, ResourceWarning)
/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/tempfile.py:817: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/srv/condor/execute/dir_27749/tmp55lzwviywandb-media'>
  _warnings.warn(warn_message, ResourceWarning)
/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/tempfile.py:817: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/srv/condor/execute/dir_27749/tmprmk7lnx4wandb-media'>
  _warnings.warn(warn_message, ResourceWarning)

Error:
====> about to start train loop
Starting training!
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1129)'))': /api/5288891/envelope/
--- Logging error ---
Traceback (most recent call last):
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/logging/__init__.py", line 1086, in emit
    stream.write(msg + self.terminator)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/wandb/sdk/lib/redirect.py", line 640, in write
    self._old_write(data)
OSError: [Errno 116] Stale file handle
Call stack:
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/threading.py", line 930, in _bootstrap
    self._bootstrap_inner()
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/threading.py", line 973, in _bootstrap_inner
    self.run()
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/threading.py", line 910, in run
    self._target(*self._args, **self._kwargs)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/sentry_sdk/worker.py", line 128, in _target
    callback()
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/sentry_sdk/transport.py", line 467, in send_envelope_wrapper
    self._send_envelope(envelope)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/sentry_sdk/transport.py", line 384, in _send_envelope
    self._send_request(
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/sentry_sdk/transport.py", line 230, in _send_request
    response = self._pool.request(
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/urllib3/request.py", line 78, in request
    return self.request_encode_body(
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/urllib3/request.py", line 170, in request_encode_body
    return self.urlopen(method, url, **extra_kw)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/urllib3/poolmanager.py", line 375, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/urllib3/connectionpool.py", line 780, in urlopen
    log.warning(
Message: "Retrying (%r) after connection broken by '%r': %s"
Arguments: (Retry(total=2, connect=None, read=None, redirect=None, status=None), SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1129)')), '/api/5288891/envelope/')

Bounty
My suggestions on what might solve this are:

Figuring out a way to stop wandb logging locally or minimize the amount of logging wandb is logging locally.
Figure out what is exactly being logged and minimize the space.
have the logging work even if all the folders are being symlinked. (imho this should work out of the box)
figuring out a systematic and simple way to find where the stale file handles are coming from.

I am surprised moving everything to /shared/rsaas/miranda9/ and running experiments from there did not solve the issue.

cross:

https://community.wandb.ai/t/how-to-stop-logging-locally-but-only-save-to-wandbs-servers-and-have-wandb-work-using-soft-links/3305
https://www.reddit.com/r/learnmachinelearning/comments/ybvo73/how_to_stop_logging_locally_but_only_save_to/
gitissue: https://github.com/wandb/wandb/issues/4409

Output:
seems like the solution is to not log to weird places with symlinks but log to real paths and instead clean up the wandb local paths often to avoid disk quota errors in your HPC. Not my fav solution but gets it done :).
Wandb should fix this, the whole point of wandb is that it works out of the box and I don't have to do MLOps and I can focus on research.
likely best to see discussions here: https://github.com/wandb/wandb/issues/4409


Instruction: count how many tensor in each tensor between 2 tensors are exactly equal

Input:
Since I have learned metric 0/1 loss for multi-label classification, and I need to create that metric
so I would like to def check that could count and return how many inside tensor between two tensors are actually matched
For Example
# Only second inside tensor are exactly equal

a = torch.tensor([[1,0,1,1], [1,0,1,1],[1,0,1,1]])
b = torch.tensor([[0,0,1,1], [1,0,1,1],[1,0,1,0]])
check(a,b) -> 1

AND
# Only second and third are exactly equal
c = torch.tensor([[1,0,1,1], [1,0,1,1],[1,0,1,1]])
d = torch.tensor([[0,0,1,1], [1,0,1,1],[1,0,1,1]])

check(c,d) -> 2

My question is that how could we define function check to retrieve 2 tensors and return the following expected output, Thank you in advanced

Output:
I'm answering my question, Thanks to the combination of @Zwang
and @LudvigH advice,

There are many ways you can do this, subtracting one tensor from the
other then summing values and finding the non zero tensors, to simply
using python's built in == function. What have you tried? - Zwang

I tried one of his methods, and that's answering my question :D, and fixed bug by @Ludvigs
def check(a,b):

    assert a.size() == b.size()
    diff_tensor =  a - b                  # [1,0,0,0],[0,0,0,0],[0,0,0,0]
    diff_tensor = torch.abs(diff_tensor)  # In case summation of them are neglect to zero, Credit goes to @LudvigH
    diff_scalar = torch.sum(diff_tensor, dim= 1) # [1 , 0 , 0]

    return torch.numel(diff_scalar[diff_scalar==0])


Instruction: Removing padding from tensor

Input:
I have a tensor like:
tensor([[ 9, -1, -1],
        [ 7, -1, -1],
        [ 6,  4, -1]])

What is the most efficient way to remove the padding and get something like:
[[9], [7], [ 6,  4]])

Thanks in advance for any help you can provide.

Output: Please consider using PackedSequence in Pytorch. It's for batches with variable-length sequences, and supported by Pytorch RNN cells. You can create a PackedSequence with torch.nn.utils.rnn.pack_padded_sequence() method. Please specify padding_value to -1 since you're using -1 for paddings.


Instruction: how to call the subclasses within the base_class?

Input:
I have problem and need some help towards object oriented
I want to create an graph dataset and some subclasses about modifying the base_graph dataset
my question how to call the subclasses within the base_class in order to remove/add nodes to the base_graph?
class base_graph(Dataset):
    def __init__(self, nodes, edges):
            ---
    def random_graph(self):
            # graph = generate random graph
    def __len__(self):
            ---
    def __repr__(self):
            ---
    def __getitem__(i):
            ---
            how to **call** Modifygraph_byAdding & Modifygraph_byDeleting here
            # like randomOrder(Modifygraph_byAdding(graph), Modifygraph_byDeleting)

class Modifygraph_byAdding:
        def __init__(self, node):
            ---
        def add_sample_to_A(self, graph):
            ---

class Modifygraph_byDeleting:
        def __init__(self, node):
            ---
        def delete_node_from_A(self, node):
            ---

Output:
By "how to call Modifygraph_byAdding & Modifygraph_byDeleting" do you mean to pass instances of those classes, or using them in combination with a user-defined call operator that executes some action?
Either case, you will at least need to provide such instances, either as arguments to __getitem__, or by giving base_graph two new members consisting of these two instances.
That said, I believe your design can be improved by simply noticing that you really don't need two new classes to define what, to me, appears to be functions that act on a specific instance of a base_graph. You could simply do
class base_graph(Dataset):
    def __init__(self, nodes, edges):
        self.nodes = nodes
        self.edges = edges

    def add(self, node):
        # code to add node to self.nodes

    def delete(self, node):
        # code to delete node from self.nodes

    def random_graph(self):
            # graph = generate random graph

    def __len__(self):
            ---

    def __repr__(self):
            ---

    def __getitem__(self, node_i):
            ---
            # here you can use self.add(node_i) and self.delete(node_i)

By the way, are you constructing new base_graphs using random_graph? Depending on how simple this function is, you may just want to stick its implementation inside of the __init__ function and delete def random_graph.


Instruction: Pytorch automatically update my python after installed

Input:
I getting started with python, I installed pytorch on my new conda environment, but it update my python version from 3.9.13 to 3.10.6. I want to maintain the 3.9.13 version. is any thing I can do?
i had try
conda install python=3.9.13

but its failed

Output:
Recreate the environment, specifying the Python version at creation time.
conda env remove -n foo
conda create -n foo -c pytorch python=3.9.13 pytorch

Note that Conda does not share Python across different environments, so this specification is arbitrary.


Instruction: Functions unavailable in DiffSharp

Input:
I have successfully installed DiffSharp (along with TorchSharp, Libsharp etc.), using the following F# script:
#r "nuget: DiffSharp.Core"
#r "nuget: DiffSharp.Backends.Reference"
#r "nuget: DiffSharp.Data"
#r "nuget: DiffSharp.Backends.Torch"
#r "nuget: TorchSharp"
#r "nuget: LibTorchSharp"

open System
open DiffSharp
open DiffSharp.Data
open DiffSharp.Model
open DiffSharp.Compose
open DiffSharp.Util
open DiffSharp.Optim

The following functions however, are not available (which should be, according to the DiffSharp API reference):
dsharp.config(backend=Backend.Torch, device=Device.CPU)
dsharp.seed(1)
let x = dsharp.randn([1024; 5])

Any ideas why? Am I missing any library or open statement?

Output:
I'm not sure why this is not working, but the DiffSharp installation isntructions recommend using the DiffSharp-cpu package (or one for cuda or lite version if you have LibTorch already) and that works fine for me:
#r "nuget: DiffSharp-cpu"

open DiffSharp

dsharp.config(backend=Backend.Torch, device=Device.CPU)
dsharp.seed(1)
let x = dsharp.randn([1024; 5])

Produces:
val x: Tensor =
  tensor([[-1.5256, -0.7502, -0.6540, -1.6095, -0.1002], ...])


Instruction: what does os.environ['KMP_DUPLICATE_LIB_OK'] actually do?

Input:
I faced some issue while trying to use pytorch on jupyter (module not found). I used pip install but my kernel kept failing.
However after adding the following code within my jupyter notebook, I managed to use pytorch without issue.
However, may I know what does this code do? Especially KMP_DUPLICATE_LIB_OK.
Thank you!

Output:
It sets the environment variable KMP_DUPLICATE_LIB_OK to True. This is the same thing that happens if you run export KMP_DUPLICATE_LIB_OK=True on the command line (depending on which shell you are using).
Environment variables are a kind of "ambient" input to programs that can be used to hold general information about the environment where the program is running - hence the name - such as the current username, home directory, and PATH. (Many of these are redundant, only kept for historical reasons, and it's a pain when they get out of sync with the actual data)
Since they are "ambient" - they are available from anywhere in any child process - they can also be a convenient way to enable various hacks and workarounds such as LD_PRELOAD.
Apparently (and I found no good source for this) the specific variable KMP_DUPLICATE_LIB_OK=True tells OpenMP to not complain if it notices that two copies of OpenMP are loaded. This doesn't necessarily mean it will work, but it does mean it won't stop you from trying to make it work.


Instruction: Cannot install PyTorch with Python 3.11 (Windows)

Input:
I recently upgraded to Python 3.11 and proceeded to get my usual libraries back.
I managed to install everything back without much trouble.
However I cannot get Pytorch!
I installed everything through pip which worked fine until getting to Pytorch.
I get this error: "ERROR: Could not find a version that satisfies the requirement torch (from versions: none)
ERROR: No matching distribution found for torch"
I tried every command suggested on PyTorch.com, but I always end-up with that same exact error message.
Has anyone encountered this after switching to Python 3.11?
Thank you in advance and have a nice day.

Output:
I saw this issue on github: https://github.com/pytorch/pytorch/issues/86566 it looks like PyTorch doesn't support 3.11 yet.
Apparently there's a nightly build that you can try to use, I haven't tested it though.


Instruction: Rearrange a 5D tensor using einops

Input:
I am analyzing the implementation of the ViViT: A Video Vision Transformer. The input tensor of this model has the sahpe of (batch size, num frames, channels=3, H, W). When it comes to patch embedding, it uses the Rearrange  that is imported from from einops.layers.torch import Rearrange as follows:
  self.to_patch_embedding = nn.Sequential(
            Rearrange('b t c (h p1) (w p2) -> b t (h w) (p1 p2 c)', p1 = patch_size, p2 = patch_size),
            nn.Linear(patch_dim, dim),
        )

My question is that what will be happened to the input tensor [b, t,c,h,w] by the Rearrange function. This is because although I read the paper but it still is ambiguous for me?
Any help is welcome.

Output:
Suppose there is an input tensor of shape (32, 10, 3, 32, 32) representing (batchsize, num frames, channels, height, width).
b t c (h p1) (w p2) with p1=2 and p2=2 decomposes the tensor to (32, 10, 3, (16, 2), (16, 2))
b t (h w) (p1 p2 c) composes the decomposed the tensor to (32, 10, 32*32=1024, 2*2*3=12)


Instruction: Tokenizer can add padding without error, but data collator cannot

Input:
I'm trying to fine tune a GPT2-based model on my data using the run_clm.py example script from HuggingFace.
I have a .json data file that looks like this:
...
{"text": "some text"}
{"text": "more text"}
...

I had to change the default behavior of the script that used to concatenate input text, because all my examples are separate demonstrations that should not be concatenated:
def add_labels(example):
    example['labels'] = example['input_ids'].copy()
    return example

with training_args.main_process_first(desc="grouping texts together"):
    lm_datasets = tokenized_datasets.map(
        add_labels,
        batched=False,
        # batch_size=1,
        num_proc=data_args.preprocessing_num_workers,
        load_from_cache_file=not data_args.overwrite_cache,
        desc=f"Grouping texts in chunks of {block_size}",
    )

This essentially only adds the appropriate 'labels' field required by CLM.
However since GPT2 has a 1024-sized context-window, the examples should be padded to that length.
I can achieve this by modifying the tokenization procedure like this:
def tokenize_function(examples):
    with CaptureLogger(tok_logger) as cl:
        output = tokenizer(
            examples[text_column_name], padding='max_length') # added: padding='max_length'
        # ...

The training runs correctly.
However, I believe this should not be done by the tokenizer, but by the data collator instead. When I remove padding='max_length' from the tokenizer, I get the following error:
ValueError: Unable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True' to have batched tensors with the same length. Perhaps your features (`labels` in this case) have excessive nesting (inputs type `list` where type `int` is expected).

And also, above that:
Traceback (most recent call last):
  File "/home/jan/repos/text2task/.venv/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 716, in convert_to_tensors
    tensor = as_tensor(value)
ValueError: expected sequence of length 9 at dim 1 (got 33)

During handling of the above exception, another exception occurred:

To fix this, I have created a data collator that should do the padding:
data_collator = DataCollatorWithPadding(tokenizer, padding='max_length')

This is what is passed to the trainer. However, the above error remains.
What's going on?

Output:
I managed to fix the error but I'm really unsure about my solution, details below. Will accept a better answer.
This seems to solve it:
data_collator = DataCollatorForSeq2Seq(tokenizer, model=model, padding=True)

Found in the documentation
It seems like DataCollatorWithPadding doesn't pad the labels?
My problem is about generating an output sequence from an input sequence, so I'm guessing that using DataCollatorForSeq2Seq is what I actually want to do. However, my data does not have separate input and target columns, but a single text column (that contains a string input => target). I'm not really that this collator is intended to be used for GPT2...


Instruction: Stable Baselines3 RuntimeError: mat1 and mat2 must have the same dtype

Input:
I am trying to implement SAC with a custom environment in Stable Baselines3 and I keep getting the error in the title. The error occurs with any off policy algorithm not just SAC.
Traceback:
File "<MY PROJECT PATH>\src\main.py", line 70, in <module>
  main()
File "<MY PROJECT PATH>\src\main.py", line 66, in main
  model.learn(total_timesteps=timesteps, reset_num_timesteps=False, tb_log_name=f"sac_{num_cars}_cars")
File "<MY PROJECT PATH>\venv\lib\site-packages\stable_baselines3\sac\sac.py", line 309, in learn
  return super().learn(
File "<MY PROJECT PATH>\venv\lib\site-packages\stable_baselines3\common\off_policy_algorithm.py", line 375, in learn
  self.train(batch_size=self.batch_size, gradient_steps=gradient_steps)
File "<MY PROJECT PATH>\venv\lib\site-packages\stable_baselines3\sac\sac.py", line 256, in train
  current_q_values = self.critic(replay_data.observations, replay_data.actions)
File "<MY PROJECT PATH>\venv\lib\site-packages\torch\nn\modules\module.py", line 1190, in _call_impl
  return forward_call(*input, **kwargs)
File "<MY PROJECT PATH>\venv\lib\site-packages\stable_baselines3\common\policies.py", line 885, in forward
  return tuple(q_net(qvalue_input) for q_net in self.q_networks)
File "<MY PROJECT PATH>\venv\lib\site-packages\stable_baselines3\common\policies.py", line 885, in <genexpr>
  return tuple(q_net(qvalue_input) for q_net in self.q_networks)
File "<MY PROJECT PATH>\venv\lib\site-packages\torch\nn\modules\module.py", line 1190, in _call_impl
  return forward_call(*input, **kwargs)
File "<MY PROJECT PATH>\venv\lib\site-packages\torch\nn\modules\container.py", line 204, in forward
  input = module(input)
File "<MY PROJECT PATH>\venv\lib\site-packages\torch\nn\modules\module.py", line 1190, in _call_impl
  return forward_call(*input, **kwargs)
File "<MY PROJECT PATH>\venv\lib\site-packages\torch\nn\modules\linear.py", line 114, in forward
  return F.linear(input, self.weight, self.bias)
RuntimeError: mat1 and mat2 must have the same dtype

Action and observation spaces:
self.action_space = Box(low=-1., high=1., shape=(2,), dtype=np.float)
self.observation_space = Box(
    np.array(
        [-np.inf] * (9 * 40) + [-np.inf] * 3 + [-np.inf] * 3 + [-np.inf] * 3
        + [0.] + [0.] + [0.] + [-1.] + [0.] * 4 + [0.] * 4 + [0.] * 4,
        dtype=np.float
    ),
    np.array(
        [np.inf] * (9 * 40) + [np.inf] * 3 + [np.inf] * 3 + [np.inf] * 3
        + [np.inf] + [1.] + [1.] + [1.] + [1.] * 4 + [np.inf] * 4 + [np.inf] * 4,
        dtype=np.float
    ),
    dtype=np.float
)

Observations are returned in the step and reset methods as a numpy array of floats.
Is there something I'm missing which is causing this error? If I use one of the environments that come with gym such as pendulum it works fine which is why I think I have a problem with my custom environment.
Thanks in advance for any help and please let me know if more information is required.

Output:
Change the inputs to float32 , default the loader set the type as float64.
inputs = inputs.to(torch.float32)


Instruction: New mamba environment force torch CPU and I don't know why

Input:
When creating a new mamba (conda) environment, I only get Pytorch's CPU package. Does anyone know how to ensure/force the GPU version?
Even if, the first thing I installed is cudatoolkit it keeps getting the CPU package. I tried with version 11.6 and 11.3, nothing changed.
This is the command I used:
mamba install pytorch torchvision torchaudio -c pytorch -c conda-forge

  + pytorch                              1.13.0  py3.9_cpu_0        pytorch/win-64          145MB
  + pytorch-mutex                           1.0  cpu                pytorch/noarch            3kB
  + requests                             2.28.1  pyhd8ed1ab_1       conda-forge/noarch     Cached
  + tbb                                2021.6.0  h91493d7_1         conda-forge/win-64      178kB
  + torchaudio                           0.13.0  py39_cpu           pytorch/win-64            5MB
  + torchvision                          0.14.0  py39_cpu           pytorch/win-64            7MB

Before you ask: my GPU is available and in other environments I successfully use Pytorch with GPU.

Output:
The command you use is the one officially recommended by PyTorch documentation.
Failure seems to be down to a new version of Pytorch (1.13) having appeared on conda-forge recently and the GPU version seems to have some dependency problems (https://anaconda.org/pytorch/pytorch/files).
Unfortunately, there does not seem to be a way to specify that one wants to have the GPU version installed. The default seems to be to install the newest version, and fall back to CPU if GPU does not work. In your case, you care more about having GPU support than being on the latest Pytorch version.
The solution is to specify pytorch==1.12.1 as you remarked in a comment:
mamba install pytorch==1.12.1 torchvision torchaudio -c pytorch -c conda-forge

Here's the error I get when trying to force Pytorch 1.13 with GPU:
❯ mamba create -n testtorch pytorch=1.13=py3.10_cuda11.6_cudnn8_0 torchvision torchaudio cudatoolkit -c pytorch/noarch -c pytorch/win-64 -c conda-forge/win-64 -c conda-forge/noarch --override-channels -d
...
Looking for: ['pytorch==1.13=py3.10_cuda11.6_cudnn8_0', 'torchvision', 'torchaudio', 'cudatoolkit']

pytorch/noarch                                              Using cache
pytorch/win-64                                              Using cache
conda-forge/win-64                                          Using cache
conda-forge/noarch                                          Using cache
Encountered problems while solving:
  - nothing provides cuda 11.6.* needed by pytorch-cuda-11.6-h867d48c_0

I've made a Pytorch bug report here: https://github.com/pytorch/pytorch/issues/87991
To install version 1.13.0 with GPU support, run:
conda install pytorch=1.13.0 torchvision torchaudio pytorch-cuda=11.6 -c pytorch -c nvidia


Instruction: How to split dataset by label or each set of data; Pytorch

Input:
I have a dataset on apple images and their sugar level.
I took 6 photos of one apple for the dataset.
So an apple has 6 photos & its sugar level.
I want to split the dataset into train and validation.
I want apple images of the whole(6 photos in one set) to go into the train or validation set.
I don't know how to split in that way.
This is CSV file for dataset
Apple is the label.

Thank you in advance!

Output:
You could simply find the apple IDs and split by those instead. This could then be passed into a dataset class so that they are split across apple ids, rather than the standard approach of splitting randomly across the rows of the df.
apple_df = pd.read_csv(...)
apple_ids = apple_df['apple'].unique() #drop_duplicates() if DataFrame
apple_ids = apple_ids.sample(frac=1) #shuffle
train_val_split = int(0.9 * len(apple_ids))
train_apple_ids = apple_ids[:train_val_split]
val_apple_ids = apple_ids[train_val_split:]

class apple_dset(torch.utils.data.Dataset):
     def __init__(self,df)
          super(apple_dset,self).__init__()
          self.df = df
     def __len__(self):
          return len(self.df.index)
     def __getitem__(self,idx):
          apple = self.df.iloc[idx]
          # do loading...
          return img, label

train_apple_df = apple_df.loc[apple_df['apple'].isin([train_apple_ids])]
val_apple_df = apple_df.loc[apple_df['apple'].isin([val_apple_ids])]

train_apple_ds = apple_dset(train_apple_df)
val_apple_ds = apple_dset(val_apple_df)


Instruction: Cannot import name 'rank_zero_only' from 'pytorch_lightning.utilities.distributed'

Input:
I am using VQGAN+CLIP_(Zooming)_(z+quantize_method_with_addons).ipynb Google Repository and when I click the cell "Loading of libraries and definitions"
It sent an error :
ImportError                               Traceback (most recent call last)
<ipython-input-6-fe8fafeed45d> in <module>
     24 from omegaconf import OmegaConf
     25 from PIL import Image
---> 26 from taming.models import cond_transformer, vqgan
     27 import torch
     28 from torch import nn, optim

1 frames
/content/taming-transformers/main.py in <module>
     10 from pytorch_lightning.trainer import Trainer
     11 from pytorch_lightning.callbacks import ModelCheckpoint, Callback, LearningRateMonitor
---> 12 from pytorch_lightning.utilities.distributed import rank_zero_only
     13
     14 from taming.data.utils import custom_collate

ImportError: cannot import name 'rank_zero_only' from 'pytorch_lightning.utilities.distributed' (/usr/local/lib/python3.7/dist-packages/pytorch_lightning/utilities/distributed.py)

I don't know how to solde this problem. I don't know how to manually install Pytorch as it said "NOTE: If your import is failing due to a missing package, you can
manually install dependencies using either !pip or !apt.
To view examples of installing some common dependencies, click the
"Open Examples" button below."
Thank you in advance if you have the solution.
Inès
I triad !pip install but I may not really know where to put this cell/line of code

Output:
pytorch_lightning has recently released a new version which will throw this error (version 1.8.0.post1 released on November 2nd 2022).
https://pypi.org/project/pytorch-lightning/#history
Just install an older version of pytorch_lightning and it will work.
In my system, I ran "pip install pytorch-lightning==1.6.5", higher versions may work as well, you can check them out by clicking on the link provided above and then clicking on release history.


Instruction: Package requirements 'tensorflow~=2.10.0', 'transformers~=4.22.1', 'TorchCRF~=1.1.0' are not satisfied

Input:
I am using PyCharm 2022.2.3 (Professional Edition), Python 3.11.0 (have any problem in compatible between this version of Python with PyTorch?) , Windows 11 x64.
Microsoft Windows [Version 10.0.22621.674]
(c) Microsoft Corporation. All rights reserved.

C:\Users\donhu>python
Python 3.11.0 (main, Oct 24 2022, 18:26:48) [MSC v.1933 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>

Full source code repository https://github.com/donhuvy/vy_thesis (still have few error for setting up).
File requirement.txt
tensorflow~=2.10.0
numpy~=1.23.3
# torch==1.12.1
PyYAML~=6.0
pandas~=1.5.0
transformers~=4.22.1
TorchCRF~=1.1.0
tqdm~=4.64.1
seqeval~=1.2.2
viet-text-tools==0.1.1

Error
Package requirements 'tensorflow~=2.10.0', 'transformers~=4.22.1', 'TorchCRF~=1.1.0' are not satisfied

File train.py
import argparse
import yaml
import pandas as pd
import torch
from TorchCRF import CRF
import transformers
from data import Dataset
from engines import train_fn
import warnings

warnings.filterwarnings("ignore")
parser = argparse.ArgumentParser()
parser.add_argument("--data_file", type=str)
parser.add_argument("--hyps_file", type=str)
args = parser.parse_args()
data_file = yaml.load(open(args.data_file), Loader=yaml.FullLoader)
hyps_file = yaml.load(open(args.hyps_file), Loader=yaml.FullLoader)
train_loader = torch.utils.data.DataLoader(
    Dataset(
        df=pd.read_csv(data_file["train_df_path"]),
        tag_names=data_file["tag_names"],
        tokenizer=transformers.AutoTokenizer.from_pretrained(hyps_file["encoder"], use_fast=False),
    ),
    num_workers=hyps_file["num_workers"],
    batch_size=hyps_file["batch_size"],
    shuffle=True,
)
val_loader = torch.utils.data.DataLoader(
    Dataset(
        df=pd.read_csv(data_file["val_df_path"]),
        tag_names=data_file["tag_names"],
        tokenizer=transformers.AutoTokenizer.from_pretrained(hyps_file["encoder"], use_fast=False),
    ),
    num_workers=hyps_file["num_workers"],
    batch_size=hyps_file["batch_size"] * 2,
)
loaders = {
    "train": train_loader,
    "val": val_loader,
}
model = transformers.RobertaForTokenClassification.from_pretrained(hyps_file["encoder"],
                                                                   num_labels=data_file["num_tags"])
if hyps_file["use_crf"]:
    criterion = CRF(num_tags=data_file["num_tags"], batch_first=True)
else:
    criterion = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=float(hyps_file["lr"]))
train_fn(
    loaders, model, torch.device(hyps_file["device"]), hyps_file["device_ids"],
    criterion,
    optimizer,
    epochs=hyps_file["epochs"],
    ckp_path="../ckps/{}.pt".format(hyps_file["encoder"].split("/")[-1]),
)

Message
ERROR: Could not find a version that satisfies the requirement torch (from versions: none)
ERROR: No matching distribution found for torch
WARNING: You are using pip version 21.3.1; however, version 22.3 is available.
You should consider upgrading via the 'D:\VNU\vy_thesis\venv\Scripts\python.exe -m pip install --upgrade pip' command.

Even update pip, not resolve problem

How to fix it?

Output:
For PyCharm, take a look at https://www.jetbrains.com/help/pycharm/managing-dependencies.html#populate_dependency_files
After you've set it up, you can click Install requirement.

Open a project with the requirements file specified, a notification bar is displayed on top of any Python or requirements file opened in Editor:

Beyond PyCharm you should also learn some bits of conda and different Python environment setup: https://docs.python-guide.org/dev/virtualenvs/

First read this https://docs.python-guide.org/dev/virtualenvs/ and https://conda.io/projects/conda/en/latest/user-guide/getting-started.html, then most probably, try conda before attempting the following.
And if you're still having issues after trying the above, then it's time to consider manually updating the (possibly stale) requirements.txt, e.g.

With the requirements:
tensorflow~=2.10.0
numpy~=1.23.3
# torch==1.12.1
PyYAML~=6.0
pandas~=1.5.0
transformers~=4.22.1
TorchCRF~=1.1.0
tqdm~=4.64.1
seqeval~=1.2.2
viet-text-tools==0.1.1

We must first kind of determine which one is lowest in the food chain that requires the most dependencies.
Generic Python libraries
tqdm~=4.64.1
PyYAML~=6.0

Data related:
pandas~=1.5.0
numpy~=1.23.3

Popular Machine Learning related:
tensorflow~=2.10.0
# torch==1.12.1
transformers~=4.22.1

Less popular machine learning tools:
seqeval~=1.2.2
TorchCRF~=1.1.0
viet-text-tools==0.1.1

For the generic python tools, it should be very safe to just update them to the LATEST version possible

For the data related and machine learning tools, the library that has the most dependencies would most probably be the transformers. In that case, you can safely update transformers to the LATEST STABLE version and that should give you appropriate matching numpy, pandas, tensorflow and pytorch versions.

Up till now our new requirements.txt would look like this:
tqdm
PyYAML
transformers

Now, looking at the seqeval, it's a library that most probably work well with transformers, so adding it without specifying version should be okay

Then we have the viet-text-tools==0.1.1 tagged to a specific versions without ~=

tqdm
PyYAML
transformers
seqeval
viet-text-tools==0.1.1

Then we're left with the biggest question mark, TorchCRF, we actually have two libraries that shares similar API:

pip install pytorch-crf: https://pytorch-crf.readthedocs.io/en/stable/
pip install TorchCRF: https://pypi.org/project/TorchCRF/

Most probably you want the latter one, so given that the torch related dependencies are handled by transformers, you can try just underspecifying the version for this.
So you final requirements.txt would look like this:
tqdm
PyYAML
transformers
seqeval
viet-text-tools==0.1.1
TorchCRF

After trying the above, do a pip freeze to take note of all the actual version of the dependencies that are installed.
Then edit the versions in your requirements.txt file accordingly, you should see something like this:
tqdm
PyYAML
transformers==4.20.1
seqeval==1.2.2
viet-text-tools==0.1.1
TorchCRF==1.1.0

(Note: You might see a different set of versions on your machine depending on your setuptools/pip versions)

Q: Do I always have to do these versioning manually?
Most probably not if you're using conda and pycharm integration https://www.jetbrains.com/help/pycharm/conda-support-creating-conda-virtual-environment.html
Q: Is it normal for environment setup to be so complicated?
Not really, but if we are setting up an environment setup for code that we're not familiar with, it's a little trial and error, esp. when the requirements.txt has many flexible compatibility ~=


Instruction: Pytorch newbie, non linear regression not converging

Input:
First im new to pytorh and DL, I want to create a simple non linear regression model, but apparently is not converging, i tried to change some hyperparams without sucess. This is the code, i guess im making wrong something obvius.

    import torch

    x1 = torch.arange(1,600,1,  dtype=torch.float32).view(-1,1)
    y1 = x1*x1

    model = torch.nn.Sequential(
        torch.nn.Linear(1,500),
        torch.nn.ReLU(),
        torch.nn.Linear(500,1)
    )

    criterion = torch.nn.MSELoss()
    optimizer = torch.optim.SGD(model.parameters(), lr=0.001)

    for i in range (10000):
        y_pred = model(x1)
        loss = criterion(y_pred, y1)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if i%100 == 0:
            print(loss)

    print(model(torch.tensor([6], dtype=torch.float32)))

Output:
Possible solutions:

input normalization
batch normalization
decrease learning rate
change MSELoss to L1Loss (more stable with large errors)
change optimizer, e.g. use Adam
etc

An example:
import torch

x1 = torch.arange(1,600, 1, dtype=torch.float32).view(-1, 1)
y1 = x1*x1

model = torch.nn.Sequential(
    torch.nn.BatchNorm1d(1, affine=False),
    torch.nn.Linear(1,500),
    torch.nn.BatchNorm1d(500),
    torch.nn.ReLU(),
    torch.nn.Linear(500,1)
)

criterion = torch.nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=1e-6)

for i in range(10000):
    y_pred = model(x1)
    loss = criterion(y_pred, y1)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    if i%100 == 0:
        print(loss.item())

model.eval()
print(model(torch.tensor([[6]], dtype=torch.float32)))


Instruction: How to free GPU memory in Pytorch CUDA

Input:
I am using Colab and Pytorch CUDA for my deep learning project and faced the problem of not being able to free up the GPU. I have read some related posts here but they did not work with my problem. Please guide me on how to free up the GPU memory.
Thank you in advance

Output:
Try this:
torch.cuda.empty_cache()

or this:
with torch.no_grad():
    torch.cuda.empty_cache()


Instruction: How to convert list of dimension 0 tensors, each attached with gradient, to a dimension 1 tensor with only one gradient?

Input:
I'm trying to make some adjustments to some pre-existing code, and can get a list that looks like the following:
[tensor(0.5209, grad_fn=<SqueezeBackward0>), tensor(0.5572, grad_fn=<SqueezeBackward0>), tensor(0.7117, grad_fn=<SqueezeBackward0>), tensor(0.5730, grad_fn=<SqueezeBackward0>), tensor(0.3471, grad_fn=<SqueezeBackward0>), tensor(0.3096, grad_fn=<SqueezeBackward0>), tensor(0.2592, grad_fn=<SqueezeBackward0>), tensor(0.5197, grad_fn=<SqueezeBackward0>), tensor(0.5979, grad_fn=<SqueezeBackward0>), tensor(0.5248, grad_fn=<SqueezeBackward0>), tensor(0.4771, grad_fn=<SqueezeBackward0>), tensor(0.6718, grad_fn=<SqueezeBackward0>), tensor(0.6507, grad_fn=<SqueezeBackward0>), tensor(0.5259, grad_fn=<SqueezeBackward0>), tensor(0.7259, grad_fn=<SqueezeBackward0>), tensor(0.5728, grad_fn=<SqueezeBackward0>), tensor(0.5604, grad_fn=<SqueezeBackward0>), tensor(0.5691, grad_fn=<SqueezeBackward0>), tensor(0.4161, grad_fn=<SqueezeBackward0>), tensor(0.6565, grad_fn=<SqueezeBackward0>), tensor(0.7343, grad_fn=<SqueezeBackward0>), tensor(0.2743, grad_fn=<SqueezeBackward0>), tensor(0.5333, grad_fn=<SqueezeBackward0>), tensor(0.5636, grad_fn=<SqueezeBackward0>), tensor(0.5657, grad_fn=<SqueezeBackward0>), tensor(0.3685, grad_fn=<SqueezeBackward0>), tensor(0.5717, grad_fn=<SqueezeBackward0>), tensor(0.4868, grad_fn=<SqueezeBackward0>), tensor(0.6732, grad_fn=<SqueezeBackward0>), tensor(0.6726, grad_fn=<SqueezeBackward0>), tensor(0.3469, grad_fn=<SqueezeBackward0>), tensor(0.5695, grad_fn=<SqueezeBackward0>), tensor(0.5520, grad_fn=<SqueezeBackward0>), tensor(0.7080, grad_fn=<SqueezeBackward0>), tensor(0.3662, grad_fn=<SqueezeBackward0>), tensor(0.5197, grad_fn=<SqueezeBackward0>), tensor(0.3505, grad_fn=<SqueezeBackward0>), tensor(0.5745, grad_fn=<SqueezeBackward0>), tensor(0.6562, grad_fn=<SqueezeBackward0>), tensor(0.5050, grad_fn=<SqueezeBackward0>), tensor(0.5242, grad_fn=<SqueezeBackward0>), tensor(0.7020, grad_fn=<SqueezeBackward0>), tensor(0.4714, grad_fn=<SqueezeBackward0>), tensor(0.4119, grad_fn=<SqueezeBackward0>), tensor(0.6880, grad_fn=<SqueezeBackward0>), tensor(0.5224, grad_fn=<SqueezeBackward0>), tensor(0.7321, grad_fn=<SqueezeBackward0>), tensor(0.5861, grad_fn=<SqueezeBackward0>), tensor(0.4058, grad_fn=<SqueezeBackward0>), tensor(0.6726, grad_fn=<SqueezeBackward0>)]
As you can see, each individual entry is a tensor requiring gradient. Of course, the backpropagation does not work unless a pass in a tensor of the form tensor([a,b,c,d,..., z], grad_fn = _) but I am not sure how to convert this list of tensors with gradient to a tensor of a list with a single attached gradient. After I obtain that, my plan is to just use something like .MSELoss() on the result. Any ideas on how to do this?

Output:
Using torch.hstack, you can put all tensors into a large one with one grad_fn.
tensors = [...]  # some tensors with grad_fn
torch.hstack(tensors)  # grad_fn is CatBackward0


Instruction: Gradient is always zero in autograd.grad()

Input:
I implemented a custom loss function, which looks like this:

However, the gradient of this function is always zero and I don't understand why.
The code for the objective function:
def objective(p, output):
  x,y = p
  a = minA
  b = minB
  r = 0.1

  XA = 1/2 -1/2 * torch.tanh(100*((x - a[0])**2 + (y - a[1])**2 - (r + 0.02)**2))
  XB = 1/2 -1/2 * torch.tanh(100*((x - b[0])**2 + (y - b[1])**2 - (r + 0.02)**2))

  q = (1-XA)*((1-XB)* output + (XB))

  output_grad, _ = torch.autograd.grad(q, (x,y))
  output_grad.requires_grad_()
  q = output_grad**2

  return q

And the code for training the model (which is a simple, fully connected NN):
model = NN(input_size)
optimizer = optim.SGD(model.parameters(), lr=learning_rate)
for e in range(epochs) :
  for configuration in total:
    print("Train for configuration", configuration)
    # Training pass
    optimizer.zero_grad()

    #output is q~
    output = model(configuration)

    #loss is the objective function we defined
    loss = objective(configuration, output.item())
    loss.backward()
    optimizer.step()

I really think the problem is in the output_grad, _ = torch.autograd.grad(q, (x,y)).
(During he training, "configuration" is a point sampled from a distribution identified by the coordinates x and y).
Thanks!!
Here I provide the code on a google colab session:
Google colab

Output:
Tanh is a bounded function and converges quite quickly to 1. Your XA and XB points are defined as
XA = 1/2 - 1/2 * torch.tanh(100*(z1 + z2 - z0))
XB = 1/2 - 1/2 * torch.tanh(100*(z3 + z4 - z0))

Since z1 + z2 - z0 and z3 + z4 - z0 are rather close to 1, you will end up with an input close to 100. This means the tanh will output 1, resulting in XA and XB begin zeros. You might not want to have this 100 coefficient if you want to have non zero outputs.


Instruction: Expanding a non-singleton dimension in PyTorch, but without copying data in memory?

Input:
Say that we have a tensor s of size [a,b,c] that is not necessarily contiguous, and b>>1.
I want to expand (but not copy) it in the second dimension for n times to get a tensor of size [a,nb,c].
The issue is that I cannot find a way to do this without explicitly copying data in memory.
The ways I know to do the operation, including
s.repeat_interleave(n,dim=1)
s.unsqueeze(-2).expand(-1,-1,n,-1).contiguous().view([a,-1,c])
s.unsqueeze(-2).expand(-1,-1,n,-1).reshape([a,-1,c])

will perform the copying step, and significantly slow things down.
Anyone know a solution?
Thanks in advance!

Output:
I don't think this is possible, and here is a minimal example to illustrate my point.
Consider a torch.Tensor [1, 2, 3], which has size (3,). If we want to expand it without performing a copy, we would create a new view of the tensor. Imagine for example that we want to create a view that contains twice the values of the original tensor, i.e. [1, 2, 3, 1, 2, 3] which would have size (2*3,). But it is not possible to define such a view only playing with the strides, here is why: to step from 1 to 2, or from 2 to 3, we would need the stride value to be 1. But to step from 3 to 1, we would need the stride value to be -2, but the system of strides as implemented cannot have different values for a given axis.
I am not 100% sure that it is not possible. Maybe there exists a very clever trick by using storage_offset parameter of torch.as_strided() function, or something else. Also, Perhaps this feature will be implemented in a future version, for example if you try to set a negative stride value, you have the error
>>> torch.as_strided(torch.tensor([1, 2, 3]), size=(1,), stride=(-1,))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: as_strided: Negative strides are not supported at the moment, got strides: [-1]

indicating that this functionality could change in the future (here I used pytorch version 1.13.0).
One could argue that you could first expand without copying in a new dimension using torch.Tensor.expand(), and then flatten() the result, but this does not work, let me explain why. In the documentation of expand(), it is explained that this function returns a new view of the tensor (so this does not do any copy), and in the documentation of flatten(), it is explained that this function will try to return a view of the flattened tensor, and if not possible it will return a copy. Then let's try this method, and check the memory size of the tensor at each step using tensor.storage().nbytes():
>>> a = torch.tensor([1, 2, 3])
>>> print(a)
tensor([1, 2, 3])
>>> print(a.storage().nbytes())
24
>>> b = a.expand([2, 3])
>>> print(b)
tensor([[1, 2, 3],
        [1, 2, 3]])
>>> print(b.storage().nbytes())
24
>>> c = b.flatten()
>>> print(c)
tensor([1, 2, 3, 1, 2, 3])
>>> print(c.storage().nbytes())
48

As we can see, flatten() does not seem capable to return a view of the flattened tensor since c takes twice memory compared to a. And if the pyTorch implementation of flatten() is not capable of doing that, this probably means that indeed it is not possible to do it.


Instruction: W&B won't finish the process

Input:
I'm using pytorch_ligthning and wandb to conduct some experiments. The problem is that training will silently crash before finishing in the following way:
Epoch 997/1000
0.087
Epoch 998/1000
0.080
wandb: Waiting for W&B process to finish... (success).
Epoch 999/1000
0.108

This is how the code looks like:
        wandb_logger.watch(embnet, 'all', log_freq=100)

        #Preparing data
        data.prepare_data()

        trainer_embnet = pl.Trainer(logger=wandb_logger,
                                    callbacks=[EmbNetCallback()],
                                    reload_dataloaders_every_n_epochs=1,
                                    max_epochs=cfg_emb.trainer.max_epochs)

        trainer_embnet.fit(embnet, datamodule=data)

        wandb_logger.experiment.finish()

I have several experiments to be run sequentially, and I call finish() at the end of each one. Also on W&B screen I notice that crashed appears next to the experiment name..
EDIT:
I think I have solved the issue by adding
wandb_logger.experiment.finalize('success')

before
wandb_logger.experiment.finish()

Output: Engineer from W&B here! The fact that you get this message wandb: Waiting for W&B process to finish... (success). before the actual 1000th epoch, this means there is some error happening there. Are you training on multiple GPUs and could you also share the console log in case there is any


Instruction: PyTorch Dataloader for multiple files with sliding window

Input:
I am working on a problem where I have multiple CSVs files and I need to read those multiple CSVs one by one with a sliding window. Let’s assume that, one CSV file is having 330 data points and the window size is 32 so we should be having (10*32 = 320) and the last 10 points will be discarded.
I started making a dataset that looks like this but after spending too much time, I am not able to get it working. The current code looks like this,
class CustomDataset(Dataset):
def __init__(self, data_folder, window_size):
    self.data_folder = data_folder
    self.data_file_list = [file for file in os.listdir(data_folder)]
    print(self.data_file_list)
    self.window_size = window_size

def __len__(self):
    return len(self.data_file_list[0])

def __getitem__(self, idx):
    filename = self.data_file_list[idx]
    data, label = read_file(filename)
    return data, label

def read_file(self, filename):
    data = pd.read_csv(filename)
    data = data.drop(["file_name", "class_name"], axis = 1)
    features = data.drop(["class_no"], axis = 1)
    labels = data["class_no"]
    x = [features[index:index+self.window_size].values for index in range(0, len(features))]
    y = [labels[index:index+self.window_size].values for index in range(0, len(labels))]

    return x, y

Note: I can’t merge all these CSV files into one.
I am getting this error, TypeError: object of type 'type' has no len()

Output:
I propose the following workaround. According to this, the getitem function retrieves a specific window which belongs to a csv file and not the file itself. Towards this direction, find_num_of_windows computes the number of windows occur for a given csv file. The len(self) function will return the sum of the windows of all files. In this way, the idx input of the getitem function will no longer have an upper limit equal to the number of files. Instead the upper limit would be the number of windows of all files. The create_dataset_dict function assigns to all potential idx values the corresponding filename and window index.
Comments:

The code needs optimization. Though, I chose a simple way for easier
understanding.

I don't know how exatly the read_file function works,
so I just tried something as an example.

Hope it helps!
import csv

class CustomDataset(Dataset):
def __init__(self, data_folder, data_list_filename, window_size):
    self.data_folder = data_folder
    self.data_file_list = [file for file in os.listdir(data_folder)]
    self.window_size = window_size
    self.total_windows, self.dataset_dict = create_dataset_dict()

def find_num_of_windows(self, path_to_file):
    rows = 0
    with open(path_to_file) as csv_file:
        csv_reader = csv.reader(csv_file, delimiter=',')
        line_count = 0
        rows = sum([1 for row in csv_reader])
    windows = rows // self.window_size

def create_dataset_dict(self):
    total_windows = 0
    idx = 0
    dataset_dict = {}
    for i in range(len(self.data_file_list)):
        windows_of_current_file = find_num_of_windows(self.data_file_list[i])
        total_windows += windows_of_current_file

        for j in range(idx:idx + windows_of_current_file):
            dataset_dict[idx] = {
                "filename": self.data_file_list[i],
                "window_index": j
            }

    return total_windows, dataset_dict

def read_file(filename, w_index):
    with open(filename) as csv_file:
        csv_reader = csv.reader(csv_file, delimiter=',')
        line_idx = 0
        data = []
        labels = []
        for row in csv_reader:
            if line_idx >= w_index and line_idx < w_index + self.window_size:
                data.append(row[0])
                labels.append(row[1])

        return data, labels

def __len__(self):
    return self.total_windows

def __getitem__(self, idx):

    filename = self.dataset_dict[idx]["filename"]
    w_index = self.dataset_dict[idx]["window_index"]
    data, label = read_file(filename, w_index)
    return data, label

Example:
Assuming we have 3 CSV files - csv_1, csv_2 and csv_3
From csv_1 we can extract 2 windows, from csv_2 we can extract 3 windows and from csv_3 we can extract 1 window.
Then:

find_num_of_windows(path_to_csv_1) returns 2
find_num_of_windows(path_to_csv_2) returns 3
find_num_of_windows(path_to_csv_3) returns 1

By calling create_dataset_dict() function, the dataset dictionary is created and looks like the one below:
{
  1: {
       "filename": path_to_csv_1,
       "window_index": 1
     },
  2: {
       "filename": path_to_csv_1,
       "window_index": 2
     },
  3: {
       "filename": path_to_csv_2,
       "window_index": 1
     },
  4: {
       "filename": path_to_csv_2,
       "window_index": 2
     },
  5: {
       "filename": path_to_csv_2,
       "window_index": 3
     },
  6: {
       "filename": path_to_csv_3,
       "window_index": 1
     }
}

If we now call the getitem function using an idx in [1, 2, 3, 4, 5, 6], we can retrieve the corresponding window using the aformentioned dictionary. For example, if we give 4 as input to getitem we will retrieve the second window of file csv_2.


Instruction: PyTorch different between ones_tensor = torch.ones((2, 3,)) and ones_tensor = torch.ones(2, 3)?

Input:
In PyTorch, what is different between
ones_tensor = torch.ones((2, 3,))

and
ones_tensor = torch.ones(2, 3)

?

Output:
There's no difference between the two. As stated in the documentation,

size (int...) – a sequence of integers defining the shape of the output tensor. Can be a variable number of arguments or a collection like a list or tuple.

If you test it, they both produce the same tensor with the same shape.
tensor([[1., 1., 1.],
    [1., 1., 1.]])

torch.Size([2, 3])