In a Pytorch LSTM, does the base class take care of a hidden layer on its own, or must it be defined (additional questions)
Let's take the following code:
'''
LSTM class
'''
import torch
import pandas as pd
import numpy as np
import torch.nn.CrossEntropyLoss
class LSTM(nn.Module):

    def __init__(self, input_size, hidden_size, num_layers):
        super (LSTM, self).__init__()
        self.hidden_size = hidden_size
        self.lstm = nn.LSTM(input_size, hidden_size, num_layers)  

    def forward(self, x):
        # receive an input, create a new hidden state, return output?
        # reset the hidden state?
        hidden = (torch.zeros(num_layers, hidden_size), torch.zeros(num_layers, hidden_size))
        x, hidden = self.lstm(x, hidden)
        
        #since our observation has several sequences, we only want the output after the last sequence of the observation'''
        x = x[:, -1]
        return x

I have several questions here, and if permitted would rather ask them all at once rather than waiting 90 minutes between singular posts.
I've seen and followed quite a few examples of LSTMs in pytorch and each example seems to treat different pieces a bit differently. Since i'm not in expert in either python, or neural networks, this has lead me to a lot of confusion.  I'll ask my question sequentially in order of how they appear in the code above.

I've seen the hidden layer both defined, zeroed, left out and ignored entirely in a few different implementations. I know what its for, but in the implementation i've produced (which is itself an amalgamation of several tutorials) the hidden layer doesn't appear to be connected to anything.  in the forward function we take a single input pass it to the hidden layer (which is first zero'd) then call self.lstm on it.    Is this the equivalent of letting lstm &quot;handle&quot; the hidden layer itself?

Will this properly produce a hidden state?

Am I correct that the optimization only occurs during the training loop?  I was using this particular tutorial as an example:
https://pytorch.org/tutorials/beginner/basics/optimization_tutorial.html


def train_loop(dataloader, model, loss_fn, optimizer):
    size = len(dataloader.dataset)
    for batch, (X, y) in enumerate(dataloader):
        # Compute prediction and loss
        pred = model(X)
        loss = loss_fn(pred, y)

        # Backpropagation
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()


in the optimizer tutorial, i assume y is the true label for the observation, is that correct?

My intention is to use cross entropy loss, as that seems like the right one to define what i'm doing with my data (the labels are not discrete, and are real positive floats with a range, and there are 3 of them), so the output size should be 3.  Given the optimizer tutorial, all i need to do is hand the loss function the output from the training step, and the correct label, and then backpropagate. Is that correct as well?


I know there's a lot here, so i'd appreciate answers to any of the questions from anyone inclined to help, even if you cannot answer all of them.  Thank you for your time.

I've had a talk with a colleague and I've been able to answer some of my own questions, so I'll post those, since it may help others.

The hidden state zero-out is NOT modifying the hidden_layer, it is zero-ing out the hidden state at the start because the cells start empty, as you'd expect in any language object-oriented.  It turns out this is unnecessary, since for quite some time the pytorch default is to zero out these values if they're not initialized manually.

This simple implementation will produce a hidden state as written.

The answer to this is yes.  We don't &quot;grade&quot; the result of network until we get to the optimization section, or, more specifically, the loss function.

y is the true label, it was not identified in the tutorial.  Also, important to note, pred is not a &quot;prediction&quot; but a pytorch object that points to the result of the network acting upon the observation that was fed in.  In other words, printing out &quot;pred&quot; would not show you a vector of values that represents a prediction

This is also correct.  Pytorch handles the &quot;distance measure&quot; between the true label and the predicted label, on its own.


