Instruction: Are the pre-trained layers of the Huggingface BERT models frozen?

Input:
I use the following classification model from Huggingface:
model = AutoModelForSequenceClassification.from_pretrained("dbmdz/bert-base-german-cased", num_labels=2).to(device)

As I understand, this adds a dense layer at the end of the pre-trained model which has 2 output nodes. But are all the pre-trained layers before that frozen? Or are they also updated when fine-tuning? I can't find information about that in the docs...
So do I still have to do something like this?:
for param in model.bert.parameters():
    param.requires_grad = False

Output:
They are not frozen. All parameters are trainable by default. You can also check that with:
for name, param in model.named_parameters():
    print(name, param.requires_grad)

Output:
bert.embeddings.word_embeddings.weight True
bert.embeddings.position_embeddings.weight True
bert.embeddings.token_type_embeddings.weight True
bert.embeddings.LayerNorm.weight True
bert.embeddings.LayerNorm.bias True
bert.encoder.layer.0.attention.self.query.weight True
bert.encoder.layer.0.attention.self.query.bias True
bert.encoder.layer.0.attention.self.key.weight True
bert.encoder.layer.0.attention.self.key.bias True
bert.encoder.layer.0.attention.self.value.weight True
bert.encoder.layer.0.attention.self.value.bias True
bert.encoder.layer.0.attention.output.dense.weight True
bert.encoder.layer.0.attention.output.dense.bias True
bert.encoder.layer.0.attention.output.LayerNorm.weight True
bert.encoder.layer.0.attention.output.LayerNorm.bias True
bert.encoder.layer.0.intermediate.dense.weight True
bert.encoder.layer.0.intermediate.dense.bias True
bert.encoder.layer.0.output.dense.weight True
bert.encoder.layer.0.output.dense.bias True
bert.encoder.layer.0.output.LayerNorm.weight True
bert.encoder.layer.0.output.LayerNorm.bias True
bert.encoder.layer.1.attention.self.query.weight True
bert.encoder.layer.1.attention.self.query.bias True
bert.encoder.layer.1.attention.self.key.weight True
bert.encoder.layer.1.attention.self.key.bias True
bert.encoder.layer.1.attention.self.value.weight True
bert.encoder.layer.1.attention.self.value.bias True
bert.encoder.layer.1.attention.output.dense.weight True
bert.encoder.layer.1.attention.output.dense.bias True
bert.encoder.layer.1.attention.output.LayerNorm.weight True
bert.encoder.layer.1.attention.output.LayerNorm.bias True
bert.encoder.layer.1.intermediate.dense.weight True
bert.encoder.layer.1.intermediate.dense.bias True
bert.encoder.layer.1.output.dense.weight True
bert.encoder.layer.1.output.dense.bias True
bert.encoder.layer.1.output.LayerNorm.weight True
bert.encoder.layer.1.output.LayerNorm.bias True
bert.encoder.layer.2.attention.self.query.weight True
bert.encoder.layer.2.attention.self.query.bias True
bert.encoder.layer.2.attention.self.key.weight True
bert.encoder.layer.2.attention.self.key.bias True
bert.encoder.layer.2.attention.self.value.weight True
bert.encoder.layer.2.attention.self.value.bias True
bert.encoder.layer.2.attention.output.dense.weight True
bert.encoder.layer.2.attention.output.dense.bias True
bert.encoder.layer.2.attention.output.LayerNorm.weight True
bert.encoder.layer.2.attention.output.LayerNorm.bias True
bert.encoder.layer.2.intermediate.dense.weight True
bert.encoder.layer.2.intermediate.dense.bias True
bert.encoder.layer.2.output.dense.weight True
bert.encoder.layer.2.output.dense.bias True
bert.encoder.layer.2.output.LayerNorm.weight True
bert.encoder.layer.2.output.LayerNorm.bias True
bert.encoder.layer.3.attention.self.query.weight True
bert.encoder.layer.3.attention.self.query.bias True
bert.encoder.layer.3.attention.self.key.weight True
bert.encoder.layer.3.attention.self.key.bias True
bert.encoder.layer.3.attention.self.value.weight True
bert.encoder.layer.3.attention.self.value.bias True
bert.encoder.layer.3.attention.output.dense.weight True
bert.encoder.layer.3.attention.output.dense.bias True
bert.encoder.layer.3.attention.output.LayerNorm.weight True
bert.encoder.layer.3.attention.output.LayerNorm.bias True
bert.encoder.layer.3.intermediate.dense.weight True
bert.encoder.layer.3.intermediate.dense.bias True
bert.encoder.layer.3.output.dense.weight True
bert.encoder.layer.3.output.dense.bias True
bert.encoder.layer.3.output.LayerNorm.weight True
bert.encoder.layer.3.output.LayerNorm.bias True
bert.encoder.layer.4.attention.self.query.weight True
bert.encoder.layer.4.attention.self.query.bias True
bert.encoder.layer.4.attention.self.key.weight True
bert.encoder.layer.4.attention.self.key.bias True
bert.encoder.layer.4.attention.self.value.weight True
bert.encoder.layer.4.attention.self.value.bias True
bert.encoder.layer.4.attention.output.dense.weight True
bert.encoder.layer.4.attention.output.dense.bias True
bert.encoder.layer.4.attention.output.LayerNorm.weight True
bert.encoder.layer.4.attention.output.LayerNorm.bias True
bert.encoder.layer.4.intermediate.dense.weight True
bert.encoder.layer.4.intermediate.dense.bias True
bert.encoder.layer.4.output.dense.weight True
bert.encoder.layer.4.output.dense.bias True
bert.encoder.layer.4.output.LayerNorm.weight True
bert.encoder.layer.4.output.LayerNorm.bias True
bert.encoder.layer.5.attention.self.query.weight True
bert.encoder.layer.5.attention.self.query.bias True
bert.encoder.layer.5.attention.self.key.weight True
bert.encoder.layer.5.attention.self.key.bias True
bert.encoder.layer.5.attention.self.value.weight True
bert.encoder.layer.5.attention.self.value.bias True
bert.encoder.layer.5.attention.output.dense.weight True
bert.encoder.layer.5.attention.output.dense.bias True
bert.encoder.layer.5.attention.output.LayerNorm.weight True
bert.encoder.layer.5.attention.output.LayerNorm.bias True
bert.encoder.layer.5.intermediate.dense.weight True
bert.encoder.layer.5.intermediate.dense.bias True
bert.encoder.layer.5.output.dense.weight True
bert.encoder.layer.5.output.dense.bias True
bert.encoder.layer.5.output.LayerNorm.weight True
bert.encoder.layer.5.output.LayerNorm.bias True
bert.encoder.layer.6.attention.self.query.weight True
bert.encoder.layer.6.attention.self.query.bias True
bert.encoder.layer.6.attention.self.key.weight True
bert.encoder.layer.6.attention.self.key.bias True
bert.encoder.layer.6.attention.self.value.weight True
bert.encoder.layer.6.attention.self.value.bias True
bert.encoder.layer.6.attention.output.dense.weight True
bert.encoder.layer.6.attention.output.dense.bias True
bert.encoder.layer.6.attention.output.LayerNorm.weight True
bert.encoder.layer.6.attention.output.LayerNorm.bias True
bert.encoder.layer.6.intermediate.dense.weight True
bert.encoder.layer.6.intermediate.dense.bias True
bert.encoder.layer.6.output.dense.weight True
bert.encoder.layer.6.output.dense.bias True
bert.encoder.layer.6.output.LayerNorm.weight True
bert.encoder.layer.6.output.LayerNorm.bias True
bert.encoder.layer.7.attention.self.query.weight True
bert.encoder.layer.7.attention.self.query.bias True
bert.encoder.layer.7.attention.self.key.weight True
bert.encoder.layer.7.attention.self.key.bias True
bert.encoder.layer.7.attention.self.value.weight True
bert.encoder.layer.7.attention.self.value.bias True
bert.encoder.layer.7.attention.output.dense.weight True
bert.encoder.layer.7.attention.output.dense.bias True
bert.encoder.layer.7.attention.output.LayerNorm.weight True
bert.encoder.layer.7.attention.output.LayerNorm.bias True
bert.encoder.layer.7.intermediate.dense.weight True
bert.encoder.layer.7.intermediate.dense.bias True
bert.encoder.layer.7.output.dense.weight True
bert.encoder.layer.7.output.dense.bias True
bert.encoder.layer.7.output.LayerNorm.weight True
bert.encoder.layer.7.output.LayerNorm.bias True
bert.encoder.layer.8.attention.self.query.weight True
bert.encoder.layer.8.attention.self.query.bias True
bert.encoder.layer.8.attention.self.key.weight True
bert.encoder.layer.8.attention.self.key.bias True
bert.encoder.layer.8.attention.self.value.weight True
bert.encoder.layer.8.attention.self.value.bias True
bert.encoder.layer.8.attention.output.dense.weight True
bert.encoder.layer.8.attention.output.dense.bias True
bert.encoder.layer.8.attention.output.LayerNorm.weight True
bert.encoder.layer.8.attention.output.LayerNorm.bias True
bert.encoder.layer.8.intermediate.dense.weight True
bert.encoder.layer.8.intermediate.dense.bias True
bert.encoder.layer.8.output.dense.weight True
bert.encoder.layer.8.output.dense.bias True
bert.encoder.layer.8.output.LayerNorm.weight True
bert.encoder.layer.8.output.LayerNorm.bias True
bert.encoder.layer.9.attention.self.query.weight True
bert.encoder.layer.9.attention.self.query.bias True
bert.encoder.layer.9.attention.self.key.weight True
bert.encoder.layer.9.attention.self.key.bias True
bert.encoder.layer.9.attention.self.value.weight True
bert.encoder.layer.9.attention.self.value.bias True
bert.encoder.layer.9.attention.output.dense.weight True
bert.encoder.layer.9.attention.output.dense.bias True
bert.encoder.layer.9.attention.output.LayerNorm.weight True
bert.encoder.layer.9.attention.output.LayerNorm.bias True
bert.encoder.layer.9.intermediate.dense.weight True
bert.encoder.layer.9.intermediate.dense.bias True
bert.encoder.layer.9.output.dense.weight True
bert.encoder.layer.9.output.dense.bias True
bert.encoder.layer.9.output.LayerNorm.weight True
bert.encoder.layer.9.output.LayerNorm.bias True
bert.encoder.layer.10.attention.self.query.weight True
bert.encoder.layer.10.attention.self.query.bias True
bert.encoder.layer.10.attention.self.key.weight True
bert.encoder.layer.10.attention.self.key.bias True
bert.encoder.layer.10.attention.self.value.weight True
bert.encoder.layer.10.attention.self.value.bias True
bert.encoder.layer.10.attention.output.dense.weight True
bert.encoder.layer.10.attention.output.dense.bias True
bert.encoder.layer.10.attention.output.LayerNorm.weight True
bert.encoder.layer.10.attention.output.LayerNorm.bias True
bert.encoder.layer.10.intermediate.dense.weight True
bert.encoder.layer.10.intermediate.dense.bias True
bert.encoder.layer.10.output.dense.weight True
bert.encoder.layer.10.output.dense.bias True
bert.encoder.layer.10.output.LayerNorm.weight True
bert.encoder.layer.10.output.LayerNorm.bias True
bert.encoder.layer.11.attention.self.query.weight True
bert.encoder.layer.11.attention.self.query.bias True
bert.encoder.layer.11.attention.self.key.weight True
bert.encoder.layer.11.attention.self.key.bias True
bert.encoder.layer.11.attention.self.value.weight True
bert.encoder.layer.11.attention.self.value.bias True
bert.encoder.layer.11.attention.output.dense.weight True
bert.encoder.layer.11.attention.output.dense.bias True
bert.encoder.layer.11.attention.output.LayerNorm.weight True
bert.encoder.layer.11.attention.output.LayerNorm.bias True
bert.encoder.layer.11.intermediate.dense.weight True
bert.encoder.layer.11.intermediate.dense.bias True
bert.encoder.layer.11.output.dense.weight True
bert.encoder.layer.11.output.dense.bias True
bert.encoder.layer.11.output.LayerNorm.weight True
bert.encoder.layer.11.output.LayerNorm.bias True
bert.pooler.dense.weight True
bert.pooler.dense.bias True
classifier.weight True
classifier.bias True


Instruction: How to turn tensor type to original text (before tokenized) in Pytorch

Input:
for example, a tensor type data below is tokenized by a kind of English tokenizer.
tensor([[ 2992,  1852,  9439,  ...,  2610,  1704, 29189],
        [ 1852,  9439,     7,  ...,  1704, 29189, 23223],
        [ 9439,     7,  2367,  ..., 29189, 23223,   838],
        ...,
        [   12,  7469, 28844,  ...,  2973,    16,    73],
        [ 7469, 28844, 28469,  ...,    16,    73,   735],
        [28844, 28469,   191,  ...,    73,   735,  4482]])

how to transform it to original English text? (using Pytorch)

Output:
The method you're looking for is tokenizer.decode, which is applied to sequences of numbers to yield the original source text. In your case, you have a batch of sentences (i.e. sequence of sequences) so you'll need to iterate the function over your tensor, i.e.
decoded = [tokenizer.decode(x) for x in xs]

where tokenizer your tokenization model and xs the tensor you want to decode.

maybe also useful:
tokenizer also provides methods convert_ids_to_tokens which does what the name suggests, and convert_tokens_to_string which merges subword tokens into words to recover the original input.


Instruction: PyTorch: Computing the norm of batched tensors

Input:
I have tensor t with shape (Batch_Size x Dims) and another tensor v with shape (Vocab_Size x Dims).  I'd like to produce a tensor d with shape (Batch_Size x Vocab_Size), such that d[i,j] = norm(t[i] - v[j]).
Doing this for a single tensor (no batches) is trivial: d = torch.norm(v - t), since t would be broadcast.  How can I do this when the tensors have batches?

Output:
Insert unitary dimensions into v and t to make them (1 x Vocab_Size x Dims) and (Batch_Size x 1 x Dims) respectively. Next, take the broadcasted difference to get a tensor of shape (Batch_Size x Vocab_Size x Dims). Pass that to torch.norm along with the optional dim=2 argument so that the norm is taken along the last dimension. This will result in the desired (Batch_Size x Vocab_Size) tensor of norms.
d = torch.norm(v.unsqueeze(0) - t.unsqueeze(1), dim=2)

Edit: As pointed out by @KonstantinosKokos in the comments, due to the broadcasting rules used by numpy and pytorch, the leading unitary dimension on v does not need to be explicit. I.e. you can use
d = torch.norm(v - t.unsqueeze(1), dim=2)


Instruction: How to define PyTorch3D plane geometry?

Input:
I'm new to PyTorch3D and I'm trying to define a (subdivided) plane.
What I've tried to do is mirror the structure I found in PyTorch3D's ico_sphere.py.
For reference these are the contents:
# Copyright (c) Meta Platforms, Inc. and affiliates.
# All rights reserved.
#
# This source code is licensed under the BSD-style license found in the
# LICENSE file in the root directory of this source tree.

import torch
from pytorch3d.ops.subdivide_meshes import SubdivideMeshes
from pytorch3d.structures.meshes import Meshes

# Vertex coordinates for a level 0 ico-sphere.
_ico_verts0 = [
    [-0.5257, 0.8507, 0.0000],
    [0.5257, 0.8507, 0.0000],
    [-0.5257, -0.8507, 0.0000],
    [0.5257, -0.8507, 0.0000],
    [0.0000, -0.5257, 0.8507],
    [0.0000, 0.5257, 0.8507],
    [0.0000, -0.5257, -0.8507],
    [0.0000, 0.5257, -0.8507],
    [0.8507, 0.0000, -0.5257],
    [0.8507, 0.0000, 0.5257],
    [-0.8507, 0.0000, -0.5257],
    [-0.8507, 0.0000, 0.5257],
]

# Faces for level 0 ico-sphere
_ico_faces0 = [
    [0, 11, 5],
    [0, 5, 1],
    [0, 1, 7],
    [0, 7, 10],
    [0, 10, 11],
    [1, 5, 9],
    [5, 11, 4],
    [11, 10, 2],
    [10, 7, 6],
    [7, 1, 8],
    [3, 9, 4],
    [3, 4, 2],
    [3, 2, 6],
    [3, 6, 8],
    [3, 8, 9],
    [4, 9, 5],
    [2, 4, 11],
    [6, 2, 10],
    [8, 6, 7],
    [9, 8, 1],
]

def ico_sphere(level: int = 0, device=None):
    """
    Create verts and faces for a unit ico-sphere, with all faces oriented
    consistently.
    Args:
        level: integer specifying the number of iterations for subdivision
               of the mesh faces. Each additional level will result in four new
               faces per face.
        device: A torch.device object on which the outputs will be allocated.
    Returns:
        Meshes object with verts and faces.
    """
    if device is None:
        device = torch.device("cpu")
    if level < 0:
        raise ValueError("level must be >= 0.")
    if level == 0:
        verts = torch.tensor(_ico_verts0, dtype=torch.float32, device=device)
        faces = torch.tensor(_ico_faces0, dtype=torch.int64, device=device)

    else:
        mesh = ico_sphere(level - 1, device)
        subdivide = SubdivideMeshes()
        mesh = subdivide(mesh)
        verts = mesh.verts_list()[0]
        verts /= verts.norm(p=2, dim=1, keepdim=True)
        faces = mesh.faces_list()[0]
    return Meshes(verts=[verts], faces=[faces])

This is my attempt of tweaking the above for a basic quad plane from 2 triangles:
import torch
from pytorch3d.ops.subdivide_meshes import SubdivideMeshes
from pytorch3d.structures.meshes import Meshes

# Vertex coordinates for a level 0 plane.
_plane_verts0 = [
    [-0.5000,-0.5000, 0.0000], # TL
    [+0.5000,-0.5000, 0.0000], # TR
    [+0.5000,+0.5000, 0.0000], # BR
    [-0.5000,+0.5000, 0.0000]  # BL
]

# Faces for level 0 plane
_plane_faces0 = [
    [2, 1, 0],
    [0, 3, 2]
]

def plane(level: int = 0, device=None):
    """
    Create verts and faces for a unit plane, with all faces oriented
    consistently.

    Args:
        level: integer specifying the number of iterations for subdivision
               of the mesh faces. Each additional level will result in four new
               faces per face.
        device: A torch.device object on which the outputs will be allocated.

    Returns:
        Meshes object with verts and faces.
    """
    if device is None:
        device = torch.device("cpu")
    if level < 0:
        raise ValueError("level must be >= 0.")
    if level == 0:
        verts = torch.tensor(_plane_verts0, dtype=torch.float32, device=device)
        faces = torch.tensor(_plane_faces0, dtype=torch.int64, device=device)

    else:
        mesh = plane(level - 1, device)
        subdivide = SubdivideMeshes()
        mesh = subdivide(mesh)
        verts = mesh.verts_list()[0]
        verts /= verts.norm(p=2, dim=1, keepdim=True)
        faces = mesh.faces_list()[0]
    return Meshes(verts=[verts], faces=[faces])

The plane with no subdivision (level=0) seems fine.
The issue I'm having is when I subdivide the plane, the result shows holes / flipped normals by the looks of it:

I've tried to change the face indices a bit (offsetting the starting point, using CW vs CCW winding order, etc.), but the result is the same, so I'm not sure if the issue with the plane geometry itself, the subdivision, or both.
(I haven't found a good way to visualise face normals. I've tried plotly's Mesh3D option, but as far as I can tell it's doing double sided rendering and I couldn't figure out how to set it's renderer to render triangles single sided only. Any tips on visually debugging are the geometry also welcome)
Update
I've tried something slightly different: copying the vertex and face from a (triangulated) plane drawn in Blender:
'''
# Blender v2.82 (sub 7) OBJ File: ''
# www.blender.org
mtllib plane.mtl
o Plane
v -1.000000 0.000000 1.000000
v 1.000000 0.000000 1.000000
v -1.000000 0.000000 -1.000000
v 1.000000 0.000000 -1.000000
vt 1.000000 0.000000
vt 0.000000 1.000000
vt 0.000000 0.000000
vt 1.000000 1.000000
vn 0.0000 1.0000 0.0000
usemtl None
s off
f 2/1/1 3/2/1 1/3/1
f 2/1/1 4/4/1 3/2/1
'''

# Vertex coordinates for a level 0 plane.
_plane_verts0 = [
    [-1.000000, 0.000000, 1.000000],
    [1.000000, 0.000000, 1.000000],
    [-1.000000, 0.000000, -1.000000],
    [1.000000, 0.000000, -1.000000],

]

# Faces for level 0 plane
_plane_faces0 = [
    [1, 2, 0],
    [1, 3, 2]
]

This didn't work either.
My current hacky workaround is to load the plane from Blender:
blender_plane = load_objs_as_meshes(['plane.obj'], device=device)

(Once it's in PyTorch3D's Meshes format I can use SubdivideMeshes as needed.)
I would like to understand what the correct face index winding is for PyTorch3D (so I can potentially define other procedural meshes).

Output:
The solution is to remove this line: verts /= verts.norm(p=2, dim=1, keepdim=True)
In more detail:
import torch
from pytorch3d.ops.subdivide_meshes import SubdivideMeshes
from pytorch3d.structures.meshes import Meshes

# Vertex coordinates for a level 0 plane.
_plane_verts0 = [
    [-0.5000, -0.5000, 0.0000],  # TL
    [+0.5000, -0.5000, 0.0000],  # TR
    [+0.5000, +0.5000, 0.0000],  # BR
    [-0.5000, +0.5000, 0.0000],  # BL
]

# Faces for level 0 plane
_plane_faces0 = [[2, 1, 0], [0, 3, 2]]

def plane(level: int = 0, device=None):
    """
    Create verts and faces for a unit plane, with all faces oriented
    consistently.
    Args:
        level: integer specifying the number of iterations for subdivision
               of the mesh faces. Each additional level will result in four new
               faces per face.
        device: A torch.device object on which the outputs will be allocated.
    Returns:
        Meshes object with verts and faces.
    """
    if device is None:
        device = torch.device("cpu")
    if level < 0:
        raise ValueError("level must be >= 0.")
    if level == 0:
        verts = torch.tensor(_plane_verts0, dtype=torch.float32, device=device)
        faces = torch.tensor(_plane_faces0, dtype=torch.int64, device=device)
    else:
        mesh = plane(level - 1, device)
        subdivide = SubdivideMeshes()
        mesh = subdivide(mesh)
        verts = mesh.verts_list()[0]
        faces = mesh.faces_list()[0]
    return Meshes(verts=[verts], faces=[faces])


Instruction: Subdivide values in a tensor

Input:
I have a PyTorch tensor that contains the labels of some samples.
I want to split each label into n_groups groups, introducing new virtual labels.
For example, for the labels:
labels = torch.as_tensor([0, 0, 0, 1, 1, 1, 2, 2, 2], dtype=torch.long)

One possible solution to subdivide each label into n_groups=2 is the following:
subdivided_labels = [0, 3, 0, 1, 4, 1, 2, 5, 2]

The constraints are the following:

There is no assumption about the order of the initial labels
The labels should be well distributed among the groups
The label ranking should be consistent among different groups, i.e., the first label 0 in the first group should be the first label also in any other group.
The following should always be true torch.equal(labels, subdivided_labels % num_classes)
It is possible that the number of groups is greater than the number of samples for a given label

The following tests should pass for the desired algorithm:
@pytest.mark.parametrize(
    "labels",
    (
        torch.randint(100, size=(50,)),
        torch.arange(100),
        torch.ones(100),
        torch.randint(100, size=(50,)).repeat(4),
        torch.arange(100).repeat(4),
        torch.ones(100).repeat(4),
        torch.randint(100, size=(50,)).repeat_interleave(4),
        torch.arange(100).repeat_interleave(4),
        torch.ones(100).repeat_interleave(4),
    ),
)
@pytest.mark.parametrize("n_groups", (1, 2, 3, 4, 5, 50, 150))
def test_subdivide_labels(labels, n_groups):
    subdivided_labels = subdivide_labels(labels, n_groups=n_groups, num_classes=100)
    assert torch.equal(labels, subdivided_labels % 100)

@pytest.mark.parametrize(
    "labels, n_groups, n_classes, expected_result",
    (
        (
            torch.tensor([0, 0, 1, 1, 2, 2]),
            2,
            3,
            torch.tensor([0, 3, 1, 4, 2, 5]),
        ),
        (
            torch.tensor([0, 0, 1, 1, 2, 2]),
            2,
            10,
            torch.tensor([0, 10, 1, 11, 2, 12]),
        ),
        (
            torch.tensor([0, 0, 1, 1, 2, 2]),
            1,
            10,
            torch.tensor([0, 0, 1, 1, 2, 2]),
        ),
        (
            torch.tensor([0, 0, 2, 2, 1, 1]),
            2,
            3,
            torch.tensor([0, 3, 2, 5, 1, 4]),
        ),
        (
            torch.tensor([0, 0, 2, 2, 1, 1]),
            30,
            3,
            torch.tensor([0, 3, 2, 5, 1, 4]),
        ),
    ),
)
def test_subdivide_labels_with_gt(labels, n_groups, n_classes, expected_result):
    subdivided_labels = subdivide_labels(labels, n_groups=n_groups, num_classes=n_classes)
    assert torch.equal(expected_result, subdivided_labels)
    assert torch.equal(labels, subdivided_labels % n_classes)

I have a non-vectorized solution:
import torch

def subdivide_labels(labels: torch.Tensor, n_groups: int, num_classes: int) -> torch.Tensor:
    """Divide each label in groups introducing virtual labels.

    Args:
        labels: the tensor containing the labels, each label should be in [0, num_classes)
        n_groups: the number of groups to create for each label
        num_classes: the number of classes

    Returns:
        a tensor with the same shape of labels, but with each label partitioned in n_groups virtual labels
    """
    unique, counts = labels.unique(
        sorted=True,
        return_counts=True,
        return_inverse=False,
    )
    virtual_labels = labels.clone().detach()
    max_range = num_classes * (torch.arange(counts.max()) % n_groups)
    for value, count in zip(unique, counts):
        virtual_labels[labels == value] = max_range[:count] + value
    return virtual_labels

labels = torch.as_tensor([0, 0, 0, 1, 1, 1, 2, 2, 2], dtype=torch.long)
subdivide_labels(labels, n_groups=2, num_classes=3)

tensor([0, 3, 0, 1, 4, 1, 2, 5, 2])

Is it possible to vectorize this algorithm?
Alternatively, are there any faster algorithms to perform the same operation?

Output:
A variation of OP's approach can be vectorized with a grouped cumcount (numpy implementation by @divakar). All tests pass, but the output is slightly different since argsort has no 'stable' option in pytorch, AFAIK.
def vector_labels(labels, n_groups, num_classes):
    counts = torch.unique(labels, return_counts=True)[1]
    idx = counts.cumsum(0)
    id_arr = torch.ones(idx[-1], dtype=torch.long)
    id_arr[0] = 0
    id_arr[idx[:-1]] = -counts[:-1] + 1
    rng = id_arr.cumsum(0)[labels.argsort().argsort()] % n_groups
    maxr = torch.arange(n_groups) * num_classes
    return maxr[rng] + labels

labels = torch.arange(100).repeat_interleave(4)
%timeit vector_labels(labels, 2, 100)
%timeit subdivide_labels(labels, 2, 100)

Output
10000 loops, best of 5: 117 µs per loop
1000 loops, best of 5: 1.6 ms per loop

This is far from the fastest algorithm. For example a trivial O(n) approach, but only CPU and needs numba to be fast with Python.
import numpy as np
import numba as nb

@nb.njit
def numpy_labels(labels, n_groups, num_classes):
    lookup = np.zeros(labels.max() + 1, np.intp)
    res = np.empty_like(labels)
    for i in range(len(labels)):
        res[i] = num_classes * lookup[labels[i]] + labels[i]
        lookup[labels[i]] = lookup[labels[i]] + 1 if lookup[labels[i]] < n_groups-1 else 0

    return res

numpy_labels(labels.numpy(), 20, 100) # compile run
%timeit torch.from_numpy(numpy_labels(labels.numpy(), 20, 100))

Output
100000 loops, best of 5: 3.63 µs per loop


Instruction: Understanding model training and evaluation in Pytorch

Input:
I am following a Pytorch code on deep learning. Where I saw model evaluation taking place within the training epoch!
Q) Should the torch.no_grad and model.eval() be out of the training epoch loop?
Q) And how to determine that, which parameter (weight) are getting optimised by the optimiser during the back-propagation?
...
for l in range(1):
    model = GTN(num_edge=A.shape[-1],
                        num_channels=num_channels,w_in = node_features.shape[1],w_out = node_dim,
                        num_class=num_classes,num_layers=num_layers,norm=norm)

    if adaptive_lr == 'false':
        optimizer = torch.optim.Adam(model.parameters(), lr=0.005, weight_decay=0.001)
    else:
        optimizer = torch.optim.Adam([{'params':model.weight},{'params':model.linear1.parameters()},{'params':model.linear2.parameters()},
                                    {"params":model.layers.parameters(), "lr":0.5}], lr=0.005, weight_decay=0.001)

    loss = nn.CrossEntropyLoss()

    # Train & Valid & Test
    best_val_loss = 10000
    best_train_loss = 10000
    best_train_f1 = 0
    best_val_f1 = 0

    for i in range(epochs):
        print('Epoch:  ',i+1)
        model.zero_grad()
        model.train()
        loss,y_train,Ws = model(A, node_features, train_node, train_target)
        train_f1 = torch.mean(f1_score(torch.argmax(y_train.detach(),dim=1), train_target, num_classes=num_classes)).cpu().numpy()
        print('Train - Loss: {}, Macro_F1: {}'.format(loss.detach().cpu().numpy(), train_f1))

        loss.backward()
        optimizer.step()
        model.eval()
        # Valid

        with torch.no_grad():
            val_loss, y_valid,_ = model.forward(A, node_features, valid_node, valid_target)
            val_f1 = torch.mean(f1_score(torch.argmax(y_valid,dim=1), valid_target, num_classes=num_classes)).cpu().numpy()

        if val_f1 > best_val_f1:
            best_val_loss = val_loss.detach().cpu().numpy()
            best_train_loss = loss.detach().cpu().numpy()
            best_train_f1 = train_f1
            best_val_f1 = val_f1

    print('---------------Best Results--------------------')
    print('Train - Loss: {}, Macro_F1: {}'.format(best_train_loss, best_train_f1))
    print('Valid - Loss: {}, Macro_F1: {}'.format(best_val_loss, best_val_f1))
    final_f1 += best_test_f1

Output:
For each epoch, you are doing train, followed by validation/test.
For validation/test you are moving the model to evaluation model
using model.eval() and then doing forward propagation with
torch.no_grad() which is correct. Again, you are moving back the
model back to train model using model.train() at the start of
train. There is no issue with the code and you are using the model
modes correctly.

In your code, if adaptive_lr if False then you are optimizing the parameters given by model.parameters() and when adaptive_lr
is True then you are optimizing:

model.weight
model.linear1.parameters()
model.linear2.parameters()
model.layers.parameters()


Instruction: How do I create a model from a state dict?

Input:
I am trying to load a checkpoint pth file from the faster_rcnn_resnet101 model which is not currently in the PyTorch model zoo.
This causes PyTorch to throw a KeyError saying that I the layers in the state dict does not match the model architecture of faster_rcnn_fpn_resnet50 that I've loaded from the model zoo.
Note: I tried posting the architecture of the faster_rcnn_fpn_resnet50 model, but I went over the character limit of 30.000
print(state_dict[model].keys()):
backbone.stem.conv1.weight
backbone.stem.conv1.norm.weight
backbone.stem.conv1.norm.bias
backbone.stem.conv1.norm.running_mean
backbone.stem.conv1.norm.running_var
backbone.res2.0.shortcut.weight
backbone.res2.0.shortcut.norm.weight
backbone.res2.0.shortcut.norm.bias
backbone.res2.0.shortcut.norm.running_mean
backbone.res2.0.shortcut.norm.running_var
backbone.res2.0.conv1.weight
backbone.res2.0.conv1.norm.weight
backbone.res2.0.conv1.norm.bias
backbone.res2.0.conv1.norm.running_mean
backbone.res2.0.conv1.norm.running_var
backbone.res2.0.conv2.weight
backbone.res2.0.conv2.norm.weight
backbone.res2.0.conv2.norm.bias
backbone.res2.0.conv2.norm.running_mean
backbone.res2.0.conv2.norm.running_var
backbone.res2.0.conv3.weight
backbone.res2.0.conv3.norm.weight
backbone.res2.0.conv3.norm.bias
backbone.res2.0.conv3.norm.running_mean
backbone.res2.0.conv3.norm.running_var
backbone.res2.1.conv1.weight
backbone.res2.1.conv1.norm.weight
backbone.res2.1.conv1.norm.bias
backbone.res2.1.conv1.norm.running_mean
backbone.res2.1.conv1.norm.running_var
backbone.res2.1.conv2.weight
backbone.res2.1.conv2.norm.weight
backbone.res2.1.conv2.norm.bias
backbone.res2.1.conv2.norm.running_mean
backbone.res2.1.conv2.norm.running_var
backbone.res2.1.conv3.weight
backbone.res2.1.conv3.norm.weight
backbone.res2.1.conv3.norm.bias
backbone.res2.1.conv3.norm.running_mean
backbone.res2.1.conv3.norm.running_var
backbone.res2.2.conv1.weight
backbone.res2.2.conv1.norm.weight
backbone.res2.2.conv1.norm.bias
backbone.res2.2.conv1.norm.running_mean
backbone.res2.2.conv1.norm.running_var
backbone.res2.2.conv2.weight
backbone.res2.2.conv2.norm.weight
backbone.res2.2.conv2.norm.bias
backbone.res2.2.conv2.norm.running_mean
backbone.res2.2.conv2.norm.running_var
backbone.res2.2.conv3.weight
backbone.res2.2.conv3.norm.weight
backbone.res2.2.conv3.norm.bias
backbone.res2.2.conv3.norm.running_mean
backbone.res2.2.conv3.norm.running_var
backbone.res3.0.shortcut.weight
backbone.res3.0.shortcut.norm.weight
backbone.res3.0.shortcut.norm.bias
backbone.res3.0.shortcut.norm.running_mean
backbone.res3.0.shortcut.norm.running_var
backbone.res3.0.conv1.weight
backbone.res3.0.conv1.norm.weight
backbone.res3.0.conv1.norm.bias
backbone.res3.0.conv1.norm.running_mean
backbone.res3.0.conv1.norm.running_var
backbone.res3.0.conv2.weight
backbone.res3.0.conv2.norm.weight
backbone.res3.0.conv2.norm.bias
backbone.res3.0.conv2.norm.running_mean
backbone.res3.0.conv2.norm.running_var
backbone.res3.0.conv3.weight
backbone.res3.0.conv3.norm.weight
backbone.res3.0.conv3.norm.bias
backbone.res3.0.conv3.norm.running_mean
backbone.res3.0.conv3.norm.running_var
backbone.res3.1.conv1.weight
backbone.res3.1.conv1.norm.weight
backbone.res3.1.conv1.norm.bias
backbone.res3.1.conv1.norm.running_mean
backbone.res3.1.conv1.norm.running_var
backbone.res3.1.conv2.weight
backbone.res3.1.conv2.norm.weight
backbone.res3.1.conv2.norm.bias
backbone.res3.1.conv2.norm.running_mean
backbone.res3.1.conv2.norm.running_var
backbone.res3.1.conv3.weight
backbone.res3.1.conv3.norm.weight
backbone.res3.1.conv3.norm.bias
backbone.res3.1.conv3.norm.running_mean
backbone.res3.1.conv3.norm.running_var
backbone.res3.2.conv1.weight
backbone.res3.2.conv1.norm.weight
backbone.res3.2.conv1.norm.bias
backbone.res3.2.conv1.norm.running_mean
backbone.res3.2.conv1.norm.running_var
backbone.res3.2.conv2.weight
backbone.res3.2.conv2.norm.weight
backbone.res3.2.conv2.norm.bias
backbone.res3.2.conv2.norm.running_mean
backbone.res3.2.conv2.norm.running_var
backbone.res3.2.conv3.weight
backbone.res3.2.conv3.norm.weight
backbone.res3.2.conv3.norm.bias
backbone.res3.2.conv3.norm.running_mean
backbone.res3.2.conv3.norm.running_var
backbone.res3.3.conv1.weight
backbone.res3.3.conv1.norm.weight
backbone.res3.3.conv1.norm.bias
backbone.res3.3.conv1.norm.running_mean
backbone.res3.3.conv1.norm.running_var
backbone.res3.3.conv2.weight
backbone.res3.3.conv2.norm.weight
backbone.res3.3.conv2.norm.bias
backbone.res3.3.conv2.norm.running_mean
backbone.res3.3.conv2.norm.running_var
backbone.res3.3.conv3.weight
backbone.res3.3.conv3.norm.weight
backbone.res3.3.conv3.norm.bias
backbone.res3.3.conv3.norm.running_mean
backbone.res3.3.conv3.norm.running_var
backbone.res4.0.shortcut.weight
backbone.res4.0.shortcut.norm.weight
backbone.res4.0.shortcut.norm.bias
backbone.res4.0.shortcut.norm.running_mean
backbone.res4.0.shortcut.norm.running_var
backbone.res4.0.conv1.weight
backbone.res4.0.conv1.norm.weight
backbone.res4.0.conv1.norm.bias
backbone.res4.0.conv1.norm.running_mean
backbone.res4.0.conv1.norm.running_var
backbone.res4.0.conv2.weight
backbone.res4.0.conv2.norm.weight
backbone.res4.0.conv2.norm.bias
backbone.res4.0.conv2.norm.running_mean
backbone.res4.0.conv2.norm.running_var
backbone.res4.0.conv3.weight
backbone.res4.0.conv3.norm.weight
backbone.res4.0.conv3.norm.bias
backbone.res4.0.conv3.norm.running_mean
backbone.res4.0.conv3.norm.running_var
backbone.res4.1.conv1.weight
backbone.res4.1.conv1.norm.weight
backbone.res4.1.conv1.norm.bias
backbone.res4.1.conv1.norm.running_mean
backbone.res4.1.conv1.norm.running_var
backbone.res4.1.conv2.weight
backbone.res4.1.conv2.norm.weight
backbone.res4.1.conv2.norm.bias
backbone.res4.1.conv2.norm.running_mean
backbone.res4.1.conv2.norm.running_var
backbone.res4.1.conv3.weight
backbone.res4.1.conv3.norm.weight
backbone.res4.1.conv3.norm.bias
backbone.res4.1.conv3.norm.running_mean
backbone.res4.1.conv3.norm.running_var
backbone.res4.2.conv1.weight
backbone.res4.2.conv1.norm.weight
backbone.res4.2.conv1.norm.bias
backbone.res4.2.conv1.norm.running_mean
backbone.res4.2.conv1.norm.running_var
backbone.res4.2.conv2.weight
backbone.res4.2.conv2.norm.weight
backbone.res4.2.conv2.norm.bias
backbone.res4.2.conv2.norm.running_mean
backbone.res4.2.conv2.norm.running_var
backbone.res4.2.conv3.weight
backbone.res4.2.conv3.norm.weight
backbone.res4.2.conv3.norm.bias
backbone.res4.2.conv3.norm.running_mean
backbone.res4.2.conv3.norm.running_var
backbone.res4.3.conv1.weight
backbone.res4.3.conv1.norm.weight
backbone.res4.3.conv1.norm.bias
backbone.res4.3.conv1.norm.running_mean
backbone.res4.3.conv1.norm.running_var
backbone.res4.3.conv2.weight
backbone.res4.3.conv2.norm.weight
backbone.res4.3.conv2.norm.bias
backbone.res4.3.conv2.norm.running_mean
backbone.res4.3.conv2.norm.running_var
backbone.res4.3.conv3.weight
backbone.res4.3.conv3.norm.weight
backbone.res4.3.conv3.norm.bias
backbone.res4.3.conv3.norm.running_mean
backbone.res4.3.conv3.norm.running_var
backbone.res4.4.conv1.weight
backbone.res4.4.conv1.norm.weight
backbone.res4.4.conv1.norm.bias
backbone.res4.4.conv1.norm.running_mean
backbone.res4.4.conv1.norm.running_var
backbone.res4.4.conv2.weight
backbone.res4.4.conv2.norm.weight
backbone.res4.4.conv2.norm.bias
backbone.res4.4.conv2.norm.running_mean
backbone.res4.4.conv2.norm.running_var
backbone.res4.4.conv3.weight
backbone.res4.4.conv3.norm.weight
backbone.res4.4.conv3.norm.bias
backbone.res4.4.conv3.norm.running_mean
backbone.res4.4.conv3.norm.running_var
backbone.res4.5.conv1.weight
backbone.res4.5.conv1.norm.weight
backbone.res4.5.conv1.norm.bias
backbone.res4.5.conv1.norm.running_mean
backbone.res4.5.conv1.norm.running_var
backbone.res4.5.conv2.weight
backbone.res4.5.conv2.norm.weight
backbone.res4.5.conv2.norm.bias
backbone.res4.5.conv2.norm.running_mean
backbone.res4.5.conv2.norm.running_var
backbone.res4.5.conv3.weight
backbone.res4.5.conv3.norm.weight
backbone.res4.5.conv3.norm.bias
backbone.res4.5.conv3.norm.running_mean
backbone.res4.5.conv3.norm.running_var
backbone.res4.6.conv1.weight
backbone.res4.6.conv1.norm.weight
backbone.res4.6.conv1.norm.bias
backbone.res4.6.conv1.norm.running_mean
backbone.res4.6.conv1.norm.running_var
backbone.res4.6.conv2.weight
backbone.res4.6.conv2.norm.weight
backbone.res4.6.conv2.norm.bias
backbone.res4.6.conv2.norm.running_mean
backbone.res4.6.conv2.norm.running_var
backbone.res4.6.conv3.weight
backbone.res4.6.conv3.norm.weight
backbone.res4.6.conv3.norm.bias
backbone.res4.6.conv3.norm.running_mean
backbone.res4.6.conv3.norm.running_var
backbone.res4.7.conv1.weight
backbone.res4.7.conv1.norm.weight
backbone.res4.7.conv1.norm.bias
backbone.res4.7.conv1.norm.running_mean
backbone.res4.7.conv1.norm.running_var
backbone.res4.7.conv2.weight
backbone.res4.7.conv2.norm.weight
backbone.res4.7.conv2.norm.bias
backbone.res4.7.conv2.norm.running_mean
backbone.res4.7.conv2.norm.running_var
backbone.res4.7.conv3.weight
backbone.res4.7.conv3.norm.weight
backbone.res4.7.conv3.norm.bias
backbone.res4.7.conv3.norm.running_mean
backbone.res4.7.conv3.norm.running_var
backbone.res4.8.conv1.weight
backbone.res4.8.conv1.norm.weight
backbone.res4.8.conv1.norm.bias
backbone.res4.8.conv1.norm.running_mean
backbone.res4.8.conv1.norm.running_var
backbone.res4.8.conv2.weight
backbone.res4.8.conv2.norm.weight
backbone.res4.8.conv2.norm.bias
backbone.res4.8.conv2.norm.running_mean
backbone.res4.8.conv2.norm.running_var
backbone.res4.8.conv3.weight
backbone.res4.8.conv3.norm.weight
backbone.res4.8.conv3.norm.bias
backbone.res4.8.conv3.norm.running_mean
backbone.res4.8.conv3.norm.running_var
backbone.res4.9.conv1.weight
backbone.res4.9.conv1.norm.weight
backbone.res4.9.conv1.norm.bias
backbone.res4.9.conv1.norm.running_mean
backbone.res4.9.conv1.norm.running_var
backbone.res4.9.conv2.weight
backbone.res4.9.conv2.norm.weight
backbone.res4.9.conv2.norm.bias
backbone.res4.9.conv2.norm.running_mean
backbone.res4.9.conv2.norm.running_var
backbone.res4.9.conv3.weight
backbone.res4.9.conv3.norm.weight
backbone.res4.9.conv3.norm.bias
backbone.res4.9.conv3.norm.running_mean
backbone.res4.9.conv3.norm.running_var
backbone.res4.10.conv1.weight
backbone.res4.10.conv1.norm.weight
backbone.res4.10.conv1.norm.bias
backbone.res4.10.conv1.norm.running_mean
backbone.res4.10.conv1.norm.running_var
backbone.res4.10.conv2.weight
backbone.res4.10.conv2.norm.weight
backbone.res4.10.conv2.norm.bias
backbone.res4.10.conv2.norm.running_mean
backbone.res4.10.conv2.norm.running_var
backbone.res4.10.conv3.weight
backbone.res4.10.conv3.norm.weight
backbone.res4.10.conv3.norm.bias
backbone.res4.10.conv3.norm.running_mean
backbone.res4.10.conv3.norm.running_var
backbone.res4.11.conv1.weight
backbone.res4.11.conv1.norm.weight
backbone.res4.11.conv1.norm.bias
backbone.res4.11.conv1.norm.running_mean
backbone.res4.11.conv1.norm.running_var
backbone.res4.11.conv2.weight
backbone.res4.11.conv2.norm.weight
backbone.res4.11.conv2.norm.bias
backbone.res4.11.conv2.norm.running_mean
backbone.res4.11.conv2.norm.running_var
backbone.res4.11.conv3.weight
backbone.res4.11.conv3.norm.weight
backbone.res4.11.conv3.norm.bias
backbone.res4.11.conv3.norm.running_mean
backbone.res4.11.conv3.norm.running_var
backbone.res4.12.conv1.weight
backbone.res4.12.conv1.norm.weight
backbone.res4.12.conv1.norm.bias
backbone.res4.12.conv1.norm.running_mean
backbone.res4.12.conv1.norm.running_var
backbone.res4.12.conv2.weight
backbone.res4.12.conv2.norm.weight
backbone.res4.12.conv2.norm.bias
backbone.res4.12.conv2.norm.running_mean
backbone.res4.12.conv2.norm.running_var
backbone.res4.12.conv3.weight
backbone.res4.12.conv3.norm.weight
backbone.res4.12.conv3.norm.bias
backbone.res4.12.conv3.norm.running_mean
backbone.res4.12.conv3.norm.running_var
backbone.res4.13.conv1.weight
backbone.res4.13.conv1.norm.weight
backbone.res4.13.conv1.norm.bias
backbone.res4.13.conv1.norm.running_mean
backbone.res4.13.conv1.norm.running_var
backbone.res4.13.conv2.weight
backbone.res4.13.conv2.norm.weight
backbone.res4.13.conv2.norm.bias
backbone.res4.13.conv2.norm.running_mean
backbone.res4.13.conv2.norm.running_var
backbone.res4.13.conv3.weight
backbone.res4.13.conv3.norm.weight
backbone.res4.13.conv3.norm.bias
backbone.res4.13.conv3.norm.running_mean
backbone.res4.13.conv3.norm.running_var
backbone.res4.14.conv1.weight
backbone.res4.14.conv1.norm.weight
backbone.res4.14.conv1.norm.bias
backbone.res4.14.conv1.norm.running_mean
backbone.res4.14.conv1.norm.running_var
backbone.res4.14.conv2.weight
backbone.res4.14.conv2.norm.weight
backbone.res4.14.conv2.norm.bias
backbone.res4.14.conv2.norm.running_mean
backbone.res4.14.conv2.norm.running_var
backbone.res4.14.conv3.weight
backbone.res4.14.conv3.norm.weight
backbone.res4.14.conv3.norm.bias
backbone.res4.14.conv3.norm.running_mean
backbone.res4.14.conv3.norm.running_var
backbone.res4.15.conv1.weight
backbone.res4.15.conv1.norm.weight
backbone.res4.15.conv1.norm.bias
backbone.res4.15.conv1.norm.running_mean
backbone.res4.15.conv1.norm.running_var
backbone.res4.15.conv2.weight
backbone.res4.15.conv2.norm.weight
backbone.res4.15.conv2.norm.bias
backbone.res4.15.conv2.norm.running_mean
backbone.res4.15.conv2.norm.running_var
backbone.res4.15.conv3.weight
backbone.res4.15.conv3.norm.weight
backbone.res4.15.conv3.norm.bias
backbone.res4.15.conv3.norm.running_mean
backbone.res4.15.conv3.norm.running_var
backbone.res4.16.conv1.weight
backbone.res4.16.conv1.norm.weight
backbone.res4.16.conv1.norm.bias
backbone.res4.16.conv1.norm.running_mean
backbone.res4.16.conv1.norm.running_var
backbone.res4.16.conv2.weight
backbone.res4.16.conv2.norm.weight
backbone.res4.16.conv2.norm.bias
backbone.res4.16.conv2.norm.running_mean
backbone.res4.16.conv2.norm.running_var
backbone.res4.16.conv3.weight
backbone.res4.16.conv3.norm.weight
backbone.res4.16.conv3.norm.bias
backbone.res4.16.conv3.norm.running_mean
backbone.res4.16.conv3.norm.running_var
backbone.res4.17.conv1.weight
backbone.res4.17.conv1.norm.weight
backbone.res4.17.conv1.norm.bias
backbone.res4.17.conv1.norm.running_mean
backbone.res4.17.conv1.norm.running_var
backbone.res4.17.conv2.weight
backbone.res4.17.conv2.norm.weight
backbone.res4.17.conv2.norm.bias
backbone.res4.17.conv2.norm.running_mean
backbone.res4.17.conv2.norm.running_var
backbone.res4.17.conv3.weight
backbone.res4.17.conv3.norm.weight
backbone.res4.17.conv3.norm.bias
backbone.res4.17.conv3.norm.running_mean
backbone.res4.17.conv3.norm.running_var
backbone.res4.18.conv1.weight
backbone.res4.18.conv1.norm.weight
backbone.res4.18.conv1.norm.bias
backbone.res4.18.conv1.norm.running_mean
backbone.res4.18.conv1.norm.running_var
backbone.res4.18.conv2.weight
backbone.res4.18.conv2.norm.weight
backbone.res4.18.conv2.norm.bias
backbone.res4.18.conv2.norm.running_mean
backbone.res4.18.conv2.norm.running_var
backbone.res4.18.conv3.weight
backbone.res4.18.conv3.norm.weight
backbone.res4.18.conv3.norm.bias
backbone.res4.18.conv3.norm.running_mean
backbone.res4.18.conv3.norm.running_var
backbone.res4.19.conv1.weight
backbone.res4.19.conv1.norm.weight
backbone.res4.19.conv1.norm.bias
backbone.res4.19.conv1.norm.running_mean
backbone.res4.19.conv1.norm.running_var
backbone.res4.19.conv2.weight
backbone.res4.19.conv2.norm.weight
backbone.res4.19.conv2.norm.bias
backbone.res4.19.conv2.norm.running_mean
backbone.res4.19.conv2.norm.running_var
backbone.res4.19.conv3.weight
backbone.res4.19.conv3.norm.weight
backbone.res4.19.conv3.norm.bias
backbone.res4.19.conv3.norm.running_mean
backbone.res4.19.conv3.norm.running_var
backbone.res4.20.conv1.weight
backbone.res4.20.conv1.norm.weight
backbone.res4.20.conv1.norm.bias
backbone.res4.20.conv1.norm.running_mean
backbone.res4.20.conv1.norm.running_var
backbone.res4.20.conv2.weight
backbone.res4.20.conv2.norm.weight
backbone.res4.20.conv2.norm.bias
backbone.res4.20.conv2.norm.running_mean
backbone.res4.20.conv2.norm.running_var
backbone.res4.20.conv3.weight
backbone.res4.20.conv3.norm.weight
backbone.res4.20.conv3.norm.bias
backbone.res4.20.conv3.norm.running_mean
backbone.res4.20.conv3.norm.running_var
backbone.res4.21.conv1.weight
backbone.res4.21.conv1.norm.weight
backbone.res4.21.conv1.norm.bias
backbone.res4.21.conv1.norm.running_mean
backbone.res4.21.conv1.norm.running_var
backbone.res4.21.conv2.weight
backbone.res4.21.conv2.norm.weight
backbone.res4.21.conv2.norm.bias
backbone.res4.21.conv2.norm.running_mean
backbone.res4.21.conv2.norm.running_var
backbone.res4.21.conv3.weight
backbone.res4.21.conv3.norm.weight
backbone.res4.21.conv3.norm.bias
backbone.res4.21.conv3.norm.running_mean
backbone.res4.21.conv3.norm.running_var
backbone.res4.22.conv1.weight
backbone.res4.22.conv1.norm.weight
backbone.res4.22.conv1.norm.bias
backbone.res4.22.conv1.norm.running_mean
backbone.res4.22.conv1.norm.running_var
backbone.res4.22.conv2.weight
backbone.res4.22.conv2.norm.weight
backbone.res4.22.conv2.norm.bias
backbone.res4.22.conv2.norm.running_mean
backbone.res4.22.conv2.norm.running_var
backbone.res4.22.conv3.weight
backbone.res4.22.conv3.norm.weight
backbone.res4.22.conv3.norm.bias
backbone.res4.22.conv3.norm.running_mean
backbone.res4.22.conv3.norm.running_var
proposal_generator.rpn_head.conv.weight
proposal_generator.rpn_head.conv.bias
proposal_generator.rpn_head.objectness_logits.weight
proposal_generator.rpn_head.objectness_logits.bias
proposal_generator.rpn_head.anchor_deltas.weight
proposal_generator.rpn_head.anchor_deltas.bias
roi_heads.res5.0.shortcut.weight
roi_heads.res5.0.shortcut.norm.weight
roi_heads.res5.0.shortcut.norm.bias
roi_heads.res5.0.shortcut.norm.running_mean
roi_heads.res5.0.shortcut.norm.running_var
roi_heads.res5.0.conv1.weight
roi_heads.res5.0.conv1.norm.weight
roi_heads.res5.0.conv1.norm.bias
roi_heads.res5.0.conv1.norm.running_mean
roi_heads.res5.0.conv1.norm.running_var
roi_heads.res5.0.conv2.weight
roi_heads.res5.0.conv2.norm.weight
roi_heads.res5.0.conv2.norm.bias
roi_heads.res5.0.conv2.norm.running_mean
roi_heads.res5.0.conv2.norm.running_var
roi_heads.res5.0.conv3.weight
roi_heads.res5.0.conv3.norm.weight
roi_heads.res5.0.conv3.norm.bias
roi_heads.res5.0.conv3.norm.running_mean
roi_heads.res5.0.conv3.norm.running_var
roi_heads.res5.1.conv1.weight
roi_heads.res5.1.conv1.norm.weight
roi_heads.res5.1.conv1.norm.bias
roi_heads.res5.1.conv1.norm.running_mean
roi_heads.res5.1.conv1.norm.running_var
roi_heads.res5.1.conv2.weight
roi_heads.res5.1.conv2.norm.weight
roi_heads.res5.1.conv2.norm.bias
roi_heads.res5.1.conv2.norm.running_mean
roi_heads.res5.1.conv2.norm.running_var
roi_heads.res5.1.conv3.weight
roi_heads.res5.1.conv3.norm.weight
roi_heads.res5.1.conv3.norm.bias
roi_heads.res5.1.conv3.norm.running_mean
roi_heads.res5.1.conv3.norm.running_var
roi_heads.res5.2.conv1.weight
roi_heads.res5.2.conv1.norm.weight
roi_heads.res5.2.conv1.norm.bias
roi_heads.res5.2.conv1.norm.running_mean
roi_heads.res5.2.conv1.norm.running_var
roi_heads.res5.2.conv2.weight
roi_heads.res5.2.conv2.norm.weight
roi_heads.res5.2.conv2.norm.bias
roi_heads.res5.2.conv2.norm.running_mean
roi_heads.res5.2.conv2.norm.running_var
roi_heads.res5.2.conv3.weight
roi_heads.res5.2.conv3.norm.weight
roi_heads.res5.2.conv3.norm.bias
roi_heads.res5.2.conv3.norm.running_mean
roi_heads.res5.2.conv3.norm.running_var
roi_heads.box_predictor.cls_score.weight
roi_heads.box_predictor.cls_score.bias
roi_heads.box_predictor.bbox_pred.weight
roi_heads.box_predictor.bbox_pred.bias

Is there a way to dynamically create the model architecture of the faster_rcnn_resnet101 model in torch by iterating over the model dict or is there another way?
I have tried searching for a GitHub repository with a hubconf.py for torch.hub, but to no avail.

Output:
TLDR; the short answer is you can't.
The state_dict of a nn.Module contains the module's state, but not its function. It is not possible to instantiate a model with its state dictionary alone.
You can manage to initialize some of the sub-modules based on the state dict keys and the weight shapes, but this is not guaranteed. In any case you won't be able to know the model's behaviour because this information is simply not contained in its state dictionary. Therefore you are required to have access to the forward definition of the model in order to know the forward logic, i.e. which functions are applied on the input and intermediate outputs and in which order those submodules are used.

Have a look at this minimal example showing two models with identical state dictionaries. However, their forward is different from one another:
class A(nn.Linear):
    def forward(self, x):
        return super().forward(x)

class B(nn.Linear):
    def forward(self, x):
        return super().forward(x)**2

Here, a and b are initialized and the state dict is copied from one to the other:
>>> a = A(2,1)
>>> b = B(2,1)
>>> a.load_state_dict(b.state_dict())

Both a and b have the exact same state... but actually implement two different functions!


Instruction: What is PyTorch Dataset supposed to return?

Input:
I'm trying to get PyTorch to work with DataLoader, this being said to be the easiest way to handle mini batches, which are in some cases necessary for best performance.
DataLoader wants a Dataset as input.
Most of the documentation on Dataset assumes you are working with an off-the-shelf standard data set e.g. MNIST, or at least with images, and can use existing machinery as a black box. I'm working with non-image data I'm generating myself. My best current attempt to distill the documentation about how to do that, down to a minimal test case, is:
import torch
from torch import nn
from torch.utils.data import Dataset, DataLoader

class Dataset1(Dataset):
    def __init__(self):
        pass

    def __len__(self):
        return 80

    def __getitem__(self, i):
        # actual data is blank, just to test the mechanics of Dataset
        return [0.0, 0.0, 0.0], 1.0

train_dataloader = DataLoader(Dataset1(), batch_size=8)

for X, y in train_dataloader:
    print(f"X: {X}")
    print(f"y: {y.shape} {y.dtype} {y}")
    break

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.layers = nn.Sequential(
            nn.Linear(3, 10),
            nn.ReLU(),
            nn.Linear(10, 1),
            nn.Sigmoid(),
        )

    def forward(self, x):
        return self.layers(x)

device = torch.device("cpu")
model = Net().to(device)
criterion = nn.BCELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)

for epoch in range(10):
    for X, y in train_dataloader:
        X, y = X.to(device), y.to(device)

        pred = model(X)
        loss = criterion(pred, y)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

The output of the above program is:
X: [tensor([0., 0., 0., 0., 0., 0., 0., 0.], dtype=torch.float64), tensor([0., 0., 0., 0., 0., 0., 0., 0.], dtype=torch.float64), tensor([0., 0., 0., 0., 0., 0., 0., 0.], dtype=torch.float64)]
y: torch.Size([8]) torch.float64 tensor([1., 1., 1., 1., 1., 1., 1., 1.], dtype=torch.float64)
Traceback (most recent call last):
  File "C:\ml\test_dataloader.py", line 47, in <module>
    X, y = X.to(device), y.to(device)
AttributeError: 'list' object has no attribute 'to'

In all the example code I can find, X, y = X.to(device), y.to(device) succeeds, because X is indeed a tensor (whereas it is not in my version). Now I'm trying to find out what exactly converts X to a tensor, because either the example code e.g. https://pytorch.org/tutorials/beginner/basics/quickstart_tutorial.html does not do so, or I am failing to understand how and where it does.
Does Dataset itself convert things to tensors? The answer seems to be 'sort of'.
It has converted y to a tensor, a column of the y value for every example in the batch. That much, makes sense, though it has used type float64, whereas in machine learning, we usually prefer float32. I am used to the idea that Python always represents scalars in double precision, so the conversion from double to single precision happens at the time of forming a tensor, and that this can be insured by specifying the dtype parameter. But in this case Dataset seems to have formed the tensor implicitly. Is there a place or way to specify the dtype parameter?
X is not a tensor, but a list thereof. It would make intuitive sense if it were a list of the examples in the batch, but instead of a list of 8 elements each containing 3 elements, it's the other way around. So Dataset has transposed the input data, which would make sense if it is forming a tensor to match the shape of y, but instead of making a single 2d tensor, it has made a list of 1d tensors. (And, again, in double precision.) Why? Is there a way to change this behavior?
The answer posted so far to Does pytorch Dataset.__getitem__ have to return a dict? says __getitem__ can return anything. Okay, but then how does the anything get converted to the form the training procedure requires?

Output:
The dataset instance is only tasked with returning a single element of the dataset, which can take many forms: a dict, a list, an int, a float, a tensor, etc...
But the behaviour you are seeing is actually handled by your PyTorch data loader and not by the underlying dataset. This
mechanism is called collating and its implementation is done by collate_fn. You can actually provide your own as an argument to a data.DataLoader. The default collate function is provided by PyTorch as default_collate and will handle the vast majority of cases. Please have a look at its documentation, as it gives insights on what possible use cases it can handle.
With this default collate the returned batch will take the same types as the item you returned in your dataset.
You should therefore return tensors instead of a list as @dx2-66 explained.


Instruction: How can i solve this error with Pytorch summary?

Input:
I am trying to load a DNN pytorch model using:
from torch import nn
import torch.nn.functional as F

class myDNN(nn.Module):
  def __init__(self):
    super(myDNN, self).__init__()

    # layers definition

    # first convolutional block
    self.conv1 = nn.Conv1d(in_channels=2, out_channels=8, kernel_size=7)
    self.pool1 = nn.MaxPool1d(kernel_size = 2, stride=2)

    # second convolutional block
    self.conv2 = nn.Conv1d(in_channels=8, out_channels=16, kernel_size=3)
    self.pool2 = nn.MaxPool1d(kernel_size = 2, stride=2)

    # third convolutional block
    self.conv3 = nn.Conv1d(in_channels=16, out_channels=32, kernel_size=3)
    self.pool3 = nn.MaxPool1d(kernel_size = 2, stride=2)

    # fourth convolutional block
    self.conv4 = nn.Conv1d(in_channels=32, out_channels=64, kernel_size=3)
    self.pool4 = nn.MaxPool1d(kernel_size = 2, stride=2)

    # fifth convolutional block
    self.conv5 = nn.Conv1d(in_channels=64, out_channels=128, kernel_size=3)
    self.pool5 = nn.MaxPool1d(kernel_size = 2, stride=2)

    self.flatten = nn.Flatten()
    self.drop1 = nn.Dropout(p=0.5)
    self.fc1 = nn.Linear(in_features=3200, out_features=50) #3200 is a random number,
    probably wrong
    self.drop2 = nn.Dropout(p=0.5) #dropout
    self.fc2 = nn.Linear(in_features=50, out_features=25)
    self.fc3 = nn.Linear(in_features=25, out_features=2)

  def forward(self, x, y):
    x = F.relu(self.conv1(x))
    x = self.pool1(x)
    x = F.relu(self.conv2(x))
    x = self.pool2(x)
    x = F.relu(self.conv3(x))
    x = self.pool3(x)
    x = F.relu(self.conv4(x))
    x = self.pool3(x)
    x = F.relu(self.conv5(x))
    x = self.pool5(x)

    y = F.relu(self.conv1(y))
    y = self.pool1(y)
    y = F.relu(self.conv2(y))
    y = self.pool2(y)
    y = F.relu(self.conv3(y))
    y = self.pool3(y)
    y = F.relu(self.conv4(y))
    y = self.pool3(y)
    y = F.relu(self.conv5(y))
    y = self.pool5(y)

    #flatten
    x = self.flatten(x)
    y = self.flatten(y)

    w = torch.cat(x,y,1)

    w = self.drop1(w) #dropout layer
    w = F.relu(self.fc1(w)) #layer fully connected with re lu
    w = self.drop2(w)
    w = F.relu(self.fc2(w)) #layer fully connected with re lu

    w = self.fc3(w) #layer fully connected
    out = F.log_softmax(w, dim=1)

    return out

The DNN that i am trying to reproduce is the one mentioned in this paper:
https://iopscience.iop.org/article/10.1088/2399-6528/aa83fa/pdf
I am not sure that it's correct, anyway, when I load Pytorch's summary function:
model = myDNN()
print(model)

from torchsummary import summary
if torch.cuda.is_available():
  summary(model.cuda(), input_size = [(1,892),(1,492)])
else:
  summary(model, input_size = [(1,892),(1,492)])

but i get this error:
RuntimeError                              Traceback (most recent call last)
<ipython-input-123-6846b44c144c> in <module>()
      6 from torchsummary import summary
      7 if torch.cuda.is_available():
----> 8   summary(model.cuda(), input_size = [(1,892),(1,492)])
      9 else:
     10   summary(model, input_size = [(1,892),(1,492)])

5 frames
/usr/local/lib/python3.7/dist-packages/torch/nn/modules/conv.py in _conv_forward(self, input, weight, bias)
    297                             _single(0), self.dilation, self.groups)
    298         return F.conv1d(input, weight, bias, self.stride,
--> 299                         self.padding, self.dilation, self.groups)
    300
    301     def forward(self, input: Tensor) -> Tensor:

RuntimeError: Given groups=1, weight of size [8, 2, 7], expected input[2, 1, 892] to have 2 channels, but got 1 channels instead

Please any help would be grateful, thank you

Output:
The first layer of the model expects two channels rather than one.
Simply pass the correct input shape to "summary" as follows:
summary(model, ((2, dim1),(2,dim2))

Edit: In the forward function I would do the concatenation as follows (if both model's inputs have the same shape):
w = torch.cat([x,y], dim=1)
w = self.flatten(w)

Edit:
Here is a working code using the correct implementation
from torch import nn
import torch.nn.functional as F
import torch

class myDNN(nn.Module):
  def __init__(self):
    super(myDNN, self).__init__()

    # layers definition

    # first convolutional block
    self.path1_conv1 = nn.Conv1d(in_channels=2, out_channels=8, kernel_size=7)
    self.path1_pool1 = nn.MaxPool1d(kernel_size = 2, stride=2)

    # second convolutional block
    self.path1_conv2 = nn.Conv1d(in_channels=8, out_channels=16, kernel_size=3)
    self.path1_pool2 = nn.MaxPool1d(kernel_size = 2, stride=2)

    # third convolutional block
    self.path1_conv3 = nn.Conv1d(in_channels=16, out_channels=32, kernel_size=3)
    self.path1_pool3 = nn.MaxPool1d(kernel_size = 2, stride=2)

    # fourth convolutional block
    self.path1_conv4 = nn.Conv1d(in_channels=32, out_channels=64, kernel_size=3)
    self.path1_pool4 = nn.MaxPool1d(kernel_size = 2, stride=2)

    # fifth convolutional block
    self.path1_conv5 = nn.Conv1d(in_channels=64, out_channels=128, kernel_size=3)
    self.path1_pool5 = nn.MaxPool1d(kernel_size = 2, stride=2)

    self.path2_conv1 = nn.Conv1d(in_channels=2, out_channels=8, kernel_size=7)
    self.path2_pool1 = nn.MaxPool1d(kernel_size = 2, stride=2)

    # second convolutional block
    self.path2_conv2 = nn.Conv1d(in_channels=8, out_channels=16, kernel_size=3)
    self.path2_pool2 = nn.MaxPool1d(kernel_size = 2, stride=2)

    # third convolutional block
    self.path2_conv3 = nn.Conv1d(in_channels=16, out_channels=32, kernel_size=3)
    self.path2_pool3 = nn.MaxPool1d(kernel_size = 2, stride=2)

    # fourth convolutional block
    self.path2_conv4 = nn.Conv1d(in_channels=32, out_channels=64, kernel_size=3)
    self.path2_pool4 = nn.MaxPool1d(kernel_size = 2, stride=2)

    # fifth convolutional block
    self.path2_conv5 = nn.Conv1d(in_channels=64, out_channels=128, kernel_size=3)
    self.path2_pool5 = nn.MaxPool1d(kernel_size = 2, stride=2)

    self.flatten = nn.Flatten()
    self.drop1 = nn.Dropout(p=0.5)
    self.fc1 = nn.Linear(in_features=2048, out_features=50) #3200 is a random number,probably wrong
    self.drop2 = nn.Dropout(p=0.5) #dropout
    self.fc2 = nn.Linear(in_features=50, out_features=25)
    self.fc3 = nn.Linear(in_features=25, out_features=2)

  def forward(self, x, y):
    x = F.relu(self.path1_conv1(x))
    x = self.path1_pool1(x)
    x = F.relu(self.path1_conv2(x))
    x = self.path1_pool2(x)
    x = F.relu(self.path1_conv3(x))
    x = self.path1_pool3(x)
    x = F.relu(self.path1_conv4(x))
    x = self.path1_pool3(x)
    x = F.relu(self.path1_conv5(x))
    x = self.path1_pool5(x)

    y = F.relu(self.path2_conv1(y))
    y = self.path2_pool1(y)
    y = F.relu(self.path2_conv2(y))
    y = self.path2_pool2(y)
    y = F.relu(self.path2_conv3(y))
    y = self.path2_pool3(y)
    y = F.relu(self.path2_conv4(y))
    y = self.path2_pool3(y)
    y = F.relu(self.path2_conv5(y))
    y = self.path2_pool5(y)

    #flatten
    x = self.flatten(x)
    y = self.flatten(y)

    w = torch.cat([x,y],dim=1)
    print(w.shape)
    w = self.drop1(w) #dropout layer
    w = F.relu(self.fc1(w)) #layer fully connected with re lu
    w = self.drop2(w)
    w = F.relu(self.fc2(w)) #layer fully connected with re lu

    w = self.fc3(w) #layer fully connected
    out = F.log_softmax(w, dim=1)

    return out

def main():
    model = myDNN()
    print(model)
    from torchsummary import summary
    if torch.cuda.is_available():
        summary(model.cuda(), input_size = [(2,246),(2,447)])
    else:
        summary(model, input_size = [(2,246),(2,447)])
if __name__ == '__main__':
    main()


Instruction: I can't load an small AI model with nvidia 3090 in Windows (Cuda: Out Of Memory)

Input:
I come here because i am having problems loading any size of model with a nvidia 3090 (24Gb  ram)
I just followed this video (and thousands more xD) and install everything that Jeffs said here:
Jeffs instruction to install tensorflow, cuda, etc in windows
Then I installed pytorch for cuda 11.6:
pip3 install --pre torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/nightly/cu116

I have tensoflow 2.9.1 and cuda 11.5.
When I load any model of microsoft/DialoGPT-* , the vram of the 3090 go directly to 24Gb so i go out of memory.
The code in Colab its working correctly with 16 Gb of VRAM.
I tried in windows without success so I installed Ubuntu today to check it and I have the same problem!
Except one time that it loaded correctly 10Gb (I thought that was the pytorch version) but the app wasn't doing predictions because It needed libs and then it stopped working.
btw this is the code:
gpt2bot
This is what happen with my VRAM loading DialoGPT-small:
Nvidia 3090 VRAM
How can I solve this?
Edited:
Error:
Loading the pipeline 'microsoft/DialoGPT-medium'...
2022-07-06 03:38:34.293286: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-07-06 03:38:34.709282: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 21670 MB memory:  -> device: 0, name: NVIDIA GeForce RTX 3090, pci bus id: 0000:0b:00.0, compute capability: 8.6
Loading the pipeline 'microsoft/DialogRPT-updown'...
Traceback (most recent call last):
  File "C:\Users\barbi\PycharmProjects\gpt2bot\run_bot.py", line 33, in <module>
    run_telegram_bot(**config)
  File "C:\Users\barbi\PycharmProjects\gpt2bot\gpt2bot\telegram_bot.py", line 286, in run
    TelegramBot(**kwargs).run()
    TelegramBot(**kwargs).run()
    TelegramBot(**kwargs).run()
    TelegramBot(**kwargs).run()
  File "C:\Users\barbi\PycharmProjects\gpt2bot\gpt2bot\telegram_bot.py", line 240, in __init__
    self.ranker_dict = build_ranker_dict(device=device, **prior_ranker_weights, **cond_ranker_weights)
  File "C:\Users\barbi\PycharmProjects\gpt2bot\gpt2bot\utils.py", line 210, in build_ranker_dict
    pipeline=load_pipeline('sentiment-analysis', model='microsoft/DialogRPT-updown', **kwargs),
  File "C:\Users\barbi\PycharmProjects\gpt2bot\gpt2bot\utils.py", line 166, in load_pipeline
    return transformers.pipeline(task, **kwargs)
  File "C:\Users\barbi\anaconda3\envs\tensorflow\lib\site-packages\transformers\pipelines\__init__.py", line 684, in pipeline
    return pipeline_class(model=model, framework=framework, task=task, **kwargs)
  File "C:\Users\barbi\anaconda3\envs\tensorflow\lib\site-packages\transformers\pipelines\text_classification.py", line 68, in __init__
    super().__init__(**kwargs)
  File "C:\Users\barbi\anaconda3\envs\tensorflow\lib\site-packages\transformers\pipelines\base.py", line 770, in __init__
    self.model = self.model.to(self.device)
  File "C:\Users\barbi\anaconda3\envs\tensorflow\lib\site-packages\torch\nn\modules\module.py", line 927, in to
    return self._apply(convert)
  File "C:\Users\barbi\anaconda3\envs\tensorflow\lib\site-packages\torch\nn\modules\module.py", line 579, in _apply
    module._apply(fn)
  File "C:\Users\barbi\anaconda3\envs\tensorflow\lib\site-packages\torch\nn\modules\module.py", line 579, in _apply
    module._apply(fn)
  File "C:\Users\barbi\anaconda3\envs\tensorflow\lib\site-packages\torch\nn\modules\module.py", line 579, in _apply
    module._apply(fn)
  [Previous line repeated 2 more times]
  File "C:\Users\barbi\anaconda3\envs\tensorflow\lib\site-packages\torch\nn\modules\module.py", line 602, in _apply
    param_applied = fn(param)
  File "C:\Users\barbi\anaconda3\envs\tensorflow\lib\site-packages\torch\nn\modules\module.py", line 925, in convert
    return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
RuntimeError: CUDA out of memory. Tried to allocate 12.00 MiB (GPU 0; 24.00 GiB total capacity; 2.07 GiB already allocated; 0 bytes free; 2.07 GiB reserved in tota
l by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_
CUDA_ALLOC_CONF

nvidia-smi COlab
VRAM in colab after load all the models:
VRAM

Output:
SOLVED!
The problem was that I was using Tensorflow GPU + Pytorch GPU so it colapsed the VRAM. As far as I begun to use Tensorflow CPU + Pytorch GPU, everything works correctly.


Instruction: How do you make the reported step a multiple of the logging frequency in PyTorch-Lightning, not the logging frequency minus 1?

Input:
[Warning!! pedantry inside]
I'm using PyTorch Lightning to wrap my PyTorch model, but because I'm pedantic, I am finding the logger to be frustrating in the way it reports the steps at the frequency I've asked for, minus 1:

When I set log_every_n_steps=100 in Trainer, my Tensorboard output shows my metrics at step 99, 199, 299, etc. Why not at 100, 200, 300?
When I set check_val_every_n_epoch=30 in Trainer, my console output shows progress bar goes up to epoch 29, then does a validate, leaving a trail of console outputs that report metrics after epochs 29, 59, 89, etc. Like this:

Epoch 29: 100%|█████████████████████████████| 449/449 [00:26<00:00, 17.01it/s, loss=0.642, v_num=logs]
[validation] {'roc_auc': 0.663, 'bacc': 0.662, 'f1': 0.568, 'loss': 0.633}
Epoch 59: 100%|█████████████████████████████| 449/449 [00:26<00:00, 16.94it/s, loss=0.626, v_num=logs]
[validation] {'roc_auc': 0.665, 'bacc': 0.652, 'f1': 0.548, 'loss': 0.630}
Epoch 89: 100%|█████████████████████████████| 449/449 [00:27<00:00, 16.29it/s, loss=0.624, v_num=logs]
[validation] {'roc_auc': 0.665, 'bacc': 0.652, 'f1': 0.548, 'loss': 0.627}

Am I doing something wrong? Should I simply submit a PR to PL to fix this?

Output:
You are not doing anything wrong. Python uses zero-based indexing so epoch counting starts at zero as well. If you want to change the behavior of what is being displayed you will need to override the default TQDMProgressBar and modify on_train_epoch_start to display an offsetted value. You can achieve this by:
from pytorch_lightning.callbacks.progress.tqdm_progress import convert_inf

class LitProgressBar(TQDMProgressBar):
    def init_validation_tqdm(self):
        bar = super().init_validation_tqdm()
        bar.set_description("running validation...")
        return bar
    def on_train_epoch_start(self, trainer, *_) -> None:
        total_train_batches = self.total_train_batches
        total_val_batches = self.total_val_batches
        if total_train_batches != float("inf") and total_val_batches != float("inf"):
            # val can be checked multiple times per epoch
            val_checks_per_epoch = total_train_batches // trainer.val_check_batch
            total_val_batches = total_val_batches * val_checks_per_epoch
        total_batches = total_train_batches + total_val_batches
        self.main_progress_bar.reset(convert_inf(total_batches))
        self.main_progress_bar.set_description(f"Epoch {trainer.current_epoch + 1}")

Notice the +1 in the last line of code. This will offset the epoch displayed in the progress bar. Then pass your custom bar to your trainer:
# Initialize a trainer
trainer = Trainer(
    accelerator="auto",
    devices=1 if torch.cuda.is_available() else None,  # limiting got iPython runs
    max_epochs=3,
    callbacks=[LitProgressBar()],
    log_every_n_steps=100
)

Finaly:
trainer.fit(mnist_model, train_loader)

For the first epoch this will display:
GPU available: False, used: False
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs

  | Name | Type   | Params
--------------------------------
0 | l1   | Linear | 7.9 K
--------------------------------
7.9 K     Trainable params
0         Non-trainable params
7.9 K     Total params
0.031     Total estimated model params size (MB)

Epoch 1: 17%                        160/938 [00:02<00:11, 68.93it/s, loss=1.05, v_num=4]

and not the default
Epoch 0: 17%                        160/938 [00:02<00:11, 68.93it/s, loss=1.05, v_num=4]


Instruction: How to concat resnet output and original size of input image?

Input:
I have image dataset with 17 classes. There is a significant differ in sizes of some classes eg. images in one class are on average 250x200 and in the other class 25x25.
My idea was to concat output of pretrained resnet18 and original image size, because I think it's a valuable information for classification.
To be more specific - I would like to use resnet18 but to the last layer which is
(fc): Linear(in_features=512, out_features=17, bias=True)
I would like to add also Image.Shape which might be important for better classification.
Is this a reasonable solution for this kind of problem and is there a way to do it in PyTorch?

Output:
I guess you want to use the embedding that are produced by the Adaptive Pooling layer to have a fixed output size. First you need to get rid of the last linear layer (see this post):
model = models.resnet152(pretrained=True)
newmodel = torch.nn.Sequential(*(list(model.children())[:-1]))

Then you can get the embeddings and use pytorch.cat for concatenation:
out = model(X)
emb = torch.cat([out, X.shape])


Instruction: what is difference between the following optimization method?

Input:
when i am studying RNN while running the examples on the following site,
i would like to ask one question.
https://tutorials.pytorch.kr/intermediate/char_rnn_classification_tutorial
According to the site:
1. Model
import torch.nn as nn

class RNN(nn.Module):
  def __init__(self, input_size, hidden_size, output_size):
    super(RNN, self).__init__()

    self.hidden_size = hidden_size

    self.i2h = nn.Linear(input_size + hidden_size, hidden_size)
    self.i2o = nn.Linear(input_size + hidden_size, output_size)
    self.softmax = nn.LogSoftmax(dim=1)

  def forward(self, input, hidden):
    combined = torch.cat((input, hidden), 1)
    hidden = self.i2h(combined)
    output = self.i2o(combined)
    output = self.softmax(output)
    return output, hidden

  def initHidden(self):
    return torch.zeros(1, self.hidden_size)

n_hidden = 128
rnn = RNN(n_letters, n_hidden, n_categories)

learning_rate = 0.005
criterion = nn.NLLLoss()

The rnn model with pytorch is like above code
2. Training
The problem was with this part!
According to the site, learning proceeds as follows (in this case it worked fine):
def train(category_tensor, name_tensor):
  hidden = rnn.initHidden()
  rnn.zero_grad()

  for i in range(name_tensor.size()[0]):
    output, hidden = rnn(name_tensor[i], hidden)

  loss = criterion(output, category_tensor)

  loss.backward()

  for p in rnn.parameters():
    p.data.add_(p.grad.data, alpha=-learning_rate)

  return output, loss.item()

But in the pytorch model I recently learned, learning proccess was carried out by optim.step()
So, i also tried the following method (in this case, it didn't work well):

optimizer = optim.Adam(rnn.parameters(), lr = learning_rate)

def train(category_tensor, name_tensor):
  hidden = rnn.initHidden()
  rnn.zero_grad()

  for i in range(name_tensor.size()[0]):
    output, hidden = rnn(name_tensor[i], hidden)

  loss = criterion(output, category_tensor)

  optimizer.zero_grad()
  loss.backward()
  optimizer.step()

  return output, loss.item()

But in this case the training didn't work properly
3. The result window
for epoch in range(epochs):
  ...
  loss = train(category_tensor, name_tensor)
  ...

(Result window when this method is used: it seemed learning is not progressing)

so what is difference between the above optimization method?
and what should i do if i want use optim.step()??
Thanks for reading this long post and I hope you have a good day!!

Output: You are using two different optimization methods. The first one is SGD while the 2nd is Adam.


Instruction: Why CNN model cannot fit well for a pair of random tensor?

Input:
I build a toy CNN model to fit a pair of random tensors(input_tensor & truth).
batch_size = 1
channel = 3
input_size = 128
input_tensor = torch.rand((batch_size, channel, input_size, input_size))
truth = torch.rand((batch_size, channel, input_size, input_size))
device = torch.device("cuda")

class ConvModel(nn.Module):
    def __init__(self):
        super(ConvModel, self).__init__()
        self.conv1 = nn.Conv2d(3, 57344, (3, 3), (1, 1), padding=1)
        self.conv2 = nn.Conv2d(57344, 3, (3, 3), (1, 1), padding=1)
        self.relu = nn.ReLU()
        self.sigmoid = nn.Sigmoid()

    def forward(self, input_):
        x = self.conv1(input_)
        x = self.relu(x)

        x = self.conv2(x)
        x = self.sigmoid(x)
        return x

model = ConvModel().to(device)
loss_func = nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr=1e-5)

for epoch in range(100):
    output = model(input_tensor.to(device))
    loss = loss_func(output, truth.to(device))

    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
    if (1 + epoch) % 10 == 0:
        print(loss.detach().item())

I used the above codes to generate input&output pair and trained the model, and I got loss values as follow:
    0.08877705037593842
    0.08524381369352341
    0.08396070450544357
    0.0834180936217308
    0.08318136632442474
    0.08298520743846893
    0.08282201737165451
    0.08265350759029388
    0.08248833566904068
    0.08231770992279053

I'm confused that my model almost cannot fit ONE pair of data in 100 EPOCHS. Is there any problem?
Thanks for any feedback.

Output: Note that the convolution kernel is shared spatially. You network is just like trying to map a random 7*7 matrix to a random value (7 is the size of the receptive field of the output layer), and you have 128*128 this kind of pairs (despite you have only one pair of tensor). So you network failed to overfit your dataset. Reducing the input_size may help you reduce the loss.


Instruction: Imported package searches for modules in my code

Input:
Can someone explain me what is going on here and how to prevent this?
I have a main.py with the following code:
import utils
import torch

if __name__ == "__main__":
    # Foo
    print("Foo")

    # Bar
    utils.bar()

    model = torch.hub.load("ultralytics/yolov5", "yolov5s")

I outsourced some functions into a module named utils.py:
def bar():
    print("Bar")

When I run this I get the following output:
(venv) jan@xxxxx test % python main.py
Foo
Bar
Using cache found in /Users/jan/.cache/torch/hub/ultralytics_yolov5_master
Traceback (most recent call last):
File "/Users/jan/PycharmProjects/test/main.py", line 12, in <module>
    model = torch.hub.load("ultralytics/yolov5", "yolov5s")
File "/Users/jan/PycharmProjects/test/venv/lib/python3.10/site-packages/torch/hub.py", line 540, in load
    model = _load_local(repo_or_dir, model, *args, **kwargs)
File "/Users/jan/PycharmProjects/test/venv/lib/python3.10/site-packages/torch/hub.py", line 569, in _load_local
    model = entry(*args, **kwargs)
File "/Users/jan/.cache/torch/hub/ultralytics_yolov5_master/hubconf.py", line 81, in yolov5s
    return _create('yolov5s', pretrained, channels, classes, autoshape, _verbose, device)
File "/Users/jan/.cache/torch/hub/ultralytics_yolov5_master/hubconf.py", line 31, in _create
    from models.common import AutoShape, DetectMultiBackend
File "/Users/jan/.cache/torch/hub/ultralytics_yolov5_master/models/common.py", line 24, in <module>
    from utils.dataloaders import exif_transpose, letterbox
ModuleNotFoundError: No module named 'utils.dataloaders'; 'utils' is not a package

So it seems like the torch package I imported has also a utils resource (package) and searches for a module named "utils.dataloaders". Okay. But why is it searching in my utils module? And why isn't it continuing searching in its own package if it doesn't find a matching resource in my code? And last but not least: How can I prevent this situation?
I changed import utils to import utils as ut and call my function with ut.bar() but it doesn't make any difference.
The only thing that worked is to rename my utils.py to something else but this cannot be the solution...
Thanks for your help. Cheers,
Jan

Output:
The other utils package does not belong to torch, it belongs to the yolov5 repository: /Users/jan/.cache/torch/hub/ultralytics_yolov5_master/utils.
Now, to explain the error: It seems that python would search for sys.modules first when you import utils. If you import your own utils first, it is registered in sys.modules, and python would search in your own utils for things needed for the yolov5 model. Same thing would happen if you create the model first and import your own utils later (at this time, utils of the yolov5 repo is registered in sys.modules, so you won't be able to import your own utils).
The easiest solution would be to rename your utils.py, or put it under a folder, e.g.,
your_project/
  main.py
  some_name_other_than_utils/
    utils.py

and import it as
import some_name_other_than_utils.utils as utils

Then, some_name_other_than_utils.utils is registered in sys.modules so it won't affect importing in yolov5.
Another solution is to copy the yolov5 folder as a subfolder of your project. So everything in the yolov5 folder is under another namespace, without conflicting with your files. But you may need to change some import statements, and replace torch.hub.load with your own model loading function.

In most cases, having utils in your own project shouldn't conflict with third-party software that also has utils. If you try to import torch and then check sys.modules, you can see torch.utlis, torch.nn.utils, etc., rather than just utils. The following is an example of how this can be done with relative imports:
your_project/
  main.py
  utils.py
  some_3rdparty_lib/
    __init__.py
    utils.py

In some_3rdparty_lib/__init__.py:
# some_3rdparty_lib/__init__.py
from . import utils

Then in main.py:
# main.py
import utils   # your own utils
import some_3rdparty_lib.utils   # the one under some_3rdparty_lib

Note that some_3rdparty_lib does not have to be under your project directory. You can put it anywhere as long as the path is in sys.path. The imported utils would still belong to the namespace some_3rdparty_lib.


Instruction: Gathering entries in a matrix based on a matrix of column indices (tensorflow/numpy)

Input:
A little example to demonstrate what I need
I have a question about gathering in tensorflow. Let's say I have a tensor of values (that I care about for some reason):
test1 = tf.round(5*tf.random.uniform(shape=(2,3)))

which gives me this output:
<tf.Tensor: shape=(2, 3), dtype=float32, numpy=
array([[1., 1., 2.],
       [4., 5., 0.]], dtype=float32)>

and I also have a tensor of indices column indices that I want to pick out on every row:
test_ind = tf.constant([[0,1,0,0,1],
                        [0,1,1,1,0]], dtype=tf.int64)

I want to gather this so that from the first row (0th row), I pick out items in column 0, 1, 0, 0, 1, and same for the second row.
So the output for this example should be:
<tf.Tensor: shape=(2, 5), dtype=float32, numpy=
array([[1., 1., 1., 1., 1.],
       [4., 5., 5., 5., 4.]], dtype=float32)>

My attempt
So I figured out a way to do this in general, I wrote the following function gather_matrix_indices() that will take in a tensor of values and a tensor of indices and do exactly what I specified above.
def gather_matrix_indices(input_arr, index_arr):
    row, _ = input_arr.shape

    li = []

    for i in range(row):
        li.append(tf.expand_dims(tf.gather(params=input_arr[i], indices=index_arr[i]), axis=0))

    return tf.concat(li, axis=0)

My Question
I'm just wondering, is there a way to do this using ONLY tensorflow or numpy methods? The only solution I could come up with is writing my own function that iterates through every row and gathers indices for all columns in that row. I have not had runtime issues yet but I would much rather utilize built-in tensorflow or numpy methods when possible. I've tried tf.gather before too, but I don't know if this particular case is possible with any combination of tf.gather and tf.gather_nd. If anyone has a suggestion, I would greatly appreciate it.
Edit (08/18/22)
I would like to add an edit that in PyTorch, calling torch.gather() and setting dim=1 in the arguments will do EXACTLY what I wanted in this question. So if you're familiar with both libraries, and you really need this functionality, torch.gather() can do this out of the box.

Output:
You can use gather_nd() for this. It can look a bit tricky to get this working. Let me try to explain this with shapes.
We got test1 -> [2, 3] and test_ind_col_ind -> [2, 5]. test_ind_col_ind has only column indices, but you also need row indices to use gather_nd(). To use gather_nd() with a [2,3] tensor, we need to create a test_ind -> [2, 5, 2] sized tensor. The inner most dimension of this new test_ind correspond to individual indices you want to index from test1. Here we have the inner most dimension = 2 in the format (<row index>, <col index>). In other words, looking at the shape of test_ind,
[ 2 , 5 , 2 ]
    |     |
    V     |
  (2,5)   |       <- The size of the final tensor
          V
         (2,)     <- The full index to a scalar in your input tensor

import tensorflow as tf

test1 = tf.round(5*tf.random.uniform(shape=(2,3)))
print(test1)

test_ind_col_ind = tf.constant([[0,1,0,0,1],
                        [0,1,1,1,0]], dtype=tf.int64)[:, :, tf.newaxis]

test_ind_row_ind = tf.repeat(tf.range(2, dtype=tf.int64)[:, tf.newaxis, tf.newaxis], 5, axis=1)

test_ind = tf.concat([test_ind_format, test_ind], axis=-1)

res = tf.gather_nd(indices=test_ind, params=test1)


Instruction: plt.show() not showing images

Input:
My code is as follow:
import torch
import torchvision
import torchvision.transforms as transforms

import numpy
import matplotlib
import matplotlib.pyplot as plt

torch.set_printoptions(linewidth=120)

train_set = torchvision.datasets.FashionMNIST(
    root='./data/FashionMNIST',
    train=True,
    download=True,
    transform=transforms.Compose([
        transforms.ToTensor()
    ])
)

train_loader = torch.utils.data.DataLoader(
    train_set, batch_size=10
)

sample = next(iter(train_set))
image, label = sample

plt.imshow(image.squeeze(), cmap='gray')
plt.show()
print(f"label:{label}")

I try to print an image via matploblib.pylot but nothing happens.
Plus, I'm doing this on my linux server, while the same code works quite well locally on my vscode.

Output:
As an alternative to viewing your matplolib window remotely, you can always save your plot as an image file and copy it to your local machine. This is as simple as using plt.savefig
plt.save(f'label:{label}.png')


Instruction: Save normalized tensor to png in a loop using pytorch

Input:
I am working on a GAN and cannot make it work to save images that I transformed into tensors back to “normal” pngs within a loop. The same goes for the tensors that are generated by the Generator.
I applied the following transformation for the original images I am using for the training in the GAN ( I hope i did it the right way):
transform = transforms.Compose(
[
transforms.ToPILImage(),
transforms.Resize(img_size),
transforms.CenterCrop(img_size),
transforms.ToTensor(),
transforms.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5]),
]
)

When trying to save the tensors as png images in a loop with the following code they do not come out the right way:
real_samples = next(iter(train_loader))
for i in range(4):
torchvision.utils.save_image(real_samples[i, :, :, :],
‘Real_Images/real_image{}.png’.format(i))

On the left is an example of the original image after transformation and on the right an example of the “wrongly” saved ones:

Can anyone please help me out with saving the images in the right way?

Output:
You apply normalization with mean 0.5 and std 0.5, so your images are transformed from range (0., 1.) to (-1., 1.). You should denormalize them and bring back to the original range before saving them.
In your case, simply doing
real_samples = real_samples * 0.5 + 0.5

before saving should work.


Instruction: What do BatchNorm2d's running_mean / running_var mean in PyTorch?

Input:
I'd like to know what exactly the running_mean and running_var that I can call from nn.BatchNorm2d.
Example code is here where bn means nn.BatchNorm2d.
vector = torch.cat([
    torch.mean(self.conv3.bn.running_mean).view(1), torch.std(self.conv3.bn.running_mean).view(1),
    torch.mean(self.conv3.bn.running_var).view(1), torch.std(self.conv3.bn.running_var).view(1),
    torch.mean(self.conv5.bn.running_mean).view(1), torch.std(self.conv5.bn.running_mean).view(1),
    torch.mean(self.conv5.bn.running_var).view(1), torch.std(self.conv5.bn.running_var).view(1)
])

I couldn't figure out what running_mean and running_var mean in the Pytorch official documentation and user community.
What do nn.BatchNorm2.running_mean and nn.BatchNorm2.running_var mean?

Output:
From the original Batchnorm paper:
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift,Seguey Ioffe and Christian Szegedy, ICML'2015
You can see on Algorithm 1. how to measure the statistics of a given batch.

However what is kept in memory across batches is the running stats, i.e. the statistics which are measured iteratively at each batch inference. The computation of the running mean and running variance is actually quite well explained in the documentation page of nn.BatchNorm2d:

By default, the momentum coefficient is set to 0.1, it regulates how much of the current batch statistics will affect the running statistics:

closer to 1 means the new running stat is closer to the current batch statistics, whereas

closer to 0 means the current batch stats will not contribute much to updating the new running stats.

It's worth pointing out that Batchnorm2d is applied across spatial dimensions, * in addition*, to the batch dimension of course. Given a batch of shape (b, c, h, w), it will compute the statistics across (b, h, w). This means the running statistics are shaped (c,), i.e. there are as many statistics components as there are in input channels (for both mean and variance).
Here is a minimal example:
>>> bn = nn.BatchNorm2d(10)
>>> x = torch.rand(2,10,2,2)

Since track_running_stats is set to True by default on BatchNorm2d, it will track the running stats when inferring on training mode.
The running mean and variance are initialized to zeros and ones, respectively.
>>> running_mean, running_var = torch.zeros(x.size(1)),torch.ones(x.size(1))

Let's perform inference on bn in training mode and check its running stats:
>>> bn(x)
>>> bn.running_mean, bn.running_var
(tensor([0.0650, 0.0432, 0.0373, 0.0534, 0.0476,
         0.0622, 0.0651, 0.0660, 0.0406, 0.0446]),
 tensor([0.9027, 0.9170, 0.9162, 0.9082, 0.9087,
         0.9026, 0.9136, 0.9043, 0.9126, 0.9122]))

Now let's compute those stats by hand:
>>> (1-momentum)*running_mean + momentum*xmean
tensor([[0.0650, 0.0432, 0.0373, 0.0534, 0.0476,
         0.0622, 0.0651, 0.0660, 0.0406, 0.0446]])

>>> (1-momentum)*running_var + momentum*xvar
tensor([[0.9027, 0.9170, 0.9162, 0.9082, 0.9087,
         0.9026, 0.9136, 0.9043, 0.9126, 0.9122]])


Instruction: Visualizing Information from each layer: Reconstructing images with Feature Maps

Input:
I'm interested in visualizing information from a particular layer of the model. In this instance I'm using a pytorch model for ResNet18 source code for which can be found here.
Essentially the idea is to get the information each layer has for any input image that it is being trained on, and reconstruct the input image with the information that particular Conv Layer contains for the input image with the feature maps. For example, if a convolutional layer with with Nth filter corresponding to a Dogs Ear, I'd like to be able to view which CNN layer corresponds to which attribute of the image.
A given input image vector x is encoded
in each layer of the CNN by the filter responses to that image. A layer with N distinct filters
has Nl feature maps each of size Ml, where Ml is the height times the width of the feature map.
Passing the data via the pytorch dataloaders:
data_transforms = {
    'train': transforms.Compose([
        transforms.RandomResizedCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    'val': transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
}

data_dir = '/content/drive/MyDrive/Colab Notebooks/Animal Data/'
image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x),
                                          data_transforms[x])
                  for x in ['train', 'val']}
dataloaders = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size=4,
                                             shuffle=True, num_workers=4)
              for x in ['train', 'val']}
dataset_sizes = {x: len(image_datasets[x]) for x in ['train', 'val']}
class_names = image_datasets['train'].classes

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

I'm training the model here:
def train_model(model, criterion, optimizer, scheduler, num_epochs=25):
    since = time.time()

    best_model_wts = copy.deepcopy(model.state_dict())
    best_acc = 0.0

    for epoch in range(num_epochs):
        print(f'Epoch {epoch}/{num_epochs - 1}')
        print('-' * 10)

        # Each epoch has a training and validation phase
        for phase in ['train', 'val']:
            if phase == 'train':
                model.train()  # Set model to training mode
            else:
                model.eval()   # Set model to evaluate mode

            running_loss = 0.0
            running_corrects = 0

            # Iterate over data.
            for inputs, labels in dataloaders[phase]:
                inputs = inputs.to(device)
                labels = labels.to(device)

                # zero the parameter gradients
                optimizer.zero_grad()

                # forward
                # track history if only in train
                with torch.set_grad_enabled(phase == 'train'):
                    outputs = model(inputs)
                    _, preds = torch.max(outputs, 1)
                    loss = criterion(outputs, labels)

                    # backward + optimize only if in training phase
                    if phase == 'train':
                        loss.backward()
                        optimizer.step()

                # statistics
                running_loss += loss.item() * inputs.size(0)
                running_corrects += torch.sum(preds == labels.data)
            if phase == 'train':
                scheduler.step()

            epoch_loss = running_loss / dataset_sizes[phase]
            epoch_acc = running_corrects.double() / dataset_sizes[phase]

            print(f'{phase} Loss: {epoch_loss:.4f} Acc: {epoch_acc:.4f}')

            # deep copy the model
            if phase == 'val' and epoch_acc > best_acc:
                best_acc = epoch_acc
                best_model_wts = copy.deepcopy(model.state_dict())

        print()

    time_elapsed = time.time() - since
    print(f'Training complete in {time_elapsed // 60:.0f}m {time_elapsed % 60:.0f}s')
    print(f'Best val Acc: {best_acc:4f}')

    # load best model weights
    model.load_state_dict(best_model_wts)
    return model

I understand that I likely have to work with the model itself, but I'm lost on how I'd begin doing that. Any tips, or sources are highly appreciated.

Output:
I suggest that you search and read about PyTorch hooks, you can use hooks to observe the input and the output of any layer in the network, and then you can call a function to construct what you want.
You can start by reading the documentation about it, you can find it here. The idea is that you should hook a function to a layer and this function will receive the input and the output of this layer and inside this function, you will write your code to construct what you want.


Instruction: Pytorch, how to get the parameters of my network

Input:
I have a question about getting all parameters of the network. My network is defined as follow:
activation = nn.ReLU()
class OneInputBasis(nn.Module):
    def __init__(self):
        super().__init__()

        bo_b = True
        bo_last = False

        self.l1 = nn.Linear(200, 100, bias = bo_b).to(device)
        self.l4 = nn.Linear(100, 100, bias = bo_last).to(device)

    def forward(self, v):
        v = activation ( self.l1(v) )
        v = ( self.l4(v) )

        return v

and
class node(nn.Module):
    def __init__(self):
        super().__init__()

        bo_b = True
        bo_last = False

        self.set_lay = []

        for jj in range(dim_output_space_basis):
            self.set_lay.append(OneInputBasis())

    def forward(self, v):

        w = self.set_lay[0](v)

        for ii in range(dim_output_space_basis-1):
            w = torch.cat((w, self.set_lay[ii+1](v)), dim = 1 )

        return w

and
class mesh(nn.Module):
    def __init__(self):
        super().__init__()

        bo_b = True
        bo_last = False

        self.l3 = nn.Linear(2, 100, bias = bo_b).to(device)
        self.l4 = nn.Linear(100, 100, bias = bo_b).to(device)
        self.l7 = nn.Linear(100,10, bias = bo_last).to(device)

    def forward(self, w):
        w = activation ( self.l3(w) )
        w = activation ( self.l4(w) )
        w =  ( self.l7(w) )

        return w

finally, I have
activation = nn.ReLU()
class Test(nn.Module):
    def __init__(self):
        super().__init__()

        bo_b = True
        bo_last = False

        self.top = node()
        self.bottom = mesh()

    def forward(self, v, w, y):
        v = self.top(v)
        w = self.bottom(w)
        e = torch.bmm(w ,torch.bmm(v, y))

        return e[:, :, 0]

Now I define the network:
fnn_adam = Test()

When I print the parameters of the network, as
 for p in fnn_adam.parameters():
     print(p)

I can only see the parameters associated with fnn_adam.bottom, how can I print out the parameters associated with fnn_adam.top? Are the parameters associated with .top trainable? Thank you!

Output:
Calling self.set_lay.append(OneInputBasis()) with the instantiation of node does not register the fully-connected layers
 self.l1 = nn.Linear(200, 100, bias = bo_b).to(device)
 self.l4 = nn.Linear(100, 100, bias = bo_last).to(device)

to the instance fnn_adam of class Test. This is why the respective parameters do not show up in your code above.
Without loss of generality, I chose
import torch
import torch.nn as nn
import torch.nn.functional as F

dim_output_space_basis = 2
device ='cpu'

and modified the init method of class node. The remainder of your code is perfectly fine. Please see below:
class node(nn.Module):
    def __init__(self):
        super().__init__()

        bo_b = True
        bo_last = False

        # self.set_lay = [] # Legacy
        attributeNames = ['l_btm{}'.format(i) for i in range(dim_output_space_basis)]

        for jj_index, jj in enumerate(range(dim_output_space_basis)):
            # self.set_lay.append(OneInputBasis()) # Legacy
            setattr(self, attributeNames[jj_index], OneInputBasis())

Now, the parameters register as evidenced by running fnn_adam._modules and observing its output
OrderedDict([('top',
              node(
                (l_btm0): OneInputBasis(
                  (l1): Linear(in_features=200, out_features=100, bias=True)
                  (l4): Linear(in_features=100, out_features=100, bias=False)
                )
                (l_btm1): OneInputBasis(
                  (l1): Linear(in_features=200, out_features=100, bias=True)
                  (l4): Linear(in_features=100, out_features=100, bias=False)
                )
              )),
             ('bottom',
              mesh(
                (l3): Linear(in_features=2, out_features=100, bias=True)
                (l4): Linear(in_features=100, out_features=100, bias=True)
                (l7): Linear(in_features=100, out_features=10, bias=False)
              ))])