PyTorch softmax return
I am new to PyTorch and have been following this tutorial for reinforcement learning. My environment is a custom Pacman game that does not use the gym environments. The game loop is taken care of. An object in this Pacman game allows access to the state data. I use this data to send input to my Deep Q Network.
First I change the input from a python list into a tensor so that my Deep Q Network can take it as an input. Here is how I convert my python list into a tensor:
#self.getFeatures() returns a dictionary, so I grab the values and convert it to a list
input = torch.FloatTensor(list(self.getFeatures(gameState, bestAction).values())) \
             .unsqueeze(0) \
             .to(torch.device(&quot;cuda&quot; if torch.cuda.is_available() else &quot;cpu&quot;))

Then I pass in this input to my policy Deep Q Network:
test_net = self.policy_net(input).max(1)[1].view(1, 1)

Below is my Deep Q Network:
class DQN(nn.Module):

def __init__(self, feature_size, action_size):
    super(DQN, self).__init__()
    self.input = nn.Linear(feature_size, 12)
    self.hidden1 = nn.Linear(12, 5)
    self.hidden2 = nn.Linear(5, action_size)

# Called with either one element to determine next action, or a batch
# during optimization. Returns tensor([[left0exp,right0exp]...]).
def forward(self, x):
    x = F.relu(self.input(x))
    x = F.relu(self.hidden1(x))
    x = F.softmax(self.hidden2(x), dim=1)
    return x

With input tensor([[0., 1., 1., 0., 1.]]) this test_net returns this tensor([[0]]). I don't know what to get from this. I was under the impression that softmax returned a probability of each action. I have 5 actions available in my action space. I don't know what to do with my output from test_net. I want to get an action selection from this test_net, but am getting an integer.
My questions are, should the input be in a different shape? Am I converting my python input list into a tensor correctly? I have 5 features, which are tensor([[0., 1., 1., 0., 1.]]). Should the output tensor([[0]]) be a float and not a 0?

Softmax indeed assigns a probability for each action, but you are calling .max(1)[1] after you get the results from DQN, which computes max and argmax along axis 1 (.max(1)) and selects argmax ([1]). Afterwards, you also viewed it into a (1,1) shape, that's why in the end you have a 2d tensor with only one cell, containing the index that has the largest probability given by the network.
Try calling the DQN instance directly, it will return the full softmax output.
