title: Solving "TypeError: max() received an invalid combination of arguments - got (out=NoneType, axis=NoneType, ) but expected?"
body: <p>I am working on a Deep Q-Learning Reinforcement Learning algorithm for Pommerman environment.</p>
  <p>I initially got this <code>RuntimeError: mat1 and mat2 shapes cannot be multiplied (1x30 and 201x128)</code> error when I ran the main function, but I was able to get rid of it by changing the <code>--batch</code> from <code>main.py</code> to 201 so it matches the mat1 and mat2 dimensions.</p>
  <pre><code>Episode:  1 finished, result: Lose
  Avg Episode Reward: -380.0
  Traceback (most recent call last):
    File &quot;/Users/tylerkim/Desktop/rlena/DQN/main.py&quot;, line 100, in &lt;module&gt;
      main()
    File &quot;/Users/tylerkim/Desktop/rlena/DQN/main.py&quot;, line 84, in main
      agent1.update(args.gamma, args.batch)
    File &quot;/Users/tylerkim/Desktop/rlena/DQN/agent.py&quot;, line 63, in update
      expectQ = rewards + gamma * np.max(self.forward(next_states))
    File &quot;&lt;__array_function__ internals&gt;&quot;, line 6, in amax
    File &quot;/Users/tylerkim/opt/anaconda3/envs/rlena/lib/python3.7/site-packages/numpy/core/fromnumeric.py&quot;, line 2734, in amax
      keepdims=keepdims, initial=initial, where=where)
    File &quot;/Users/tylerkim/opt/anaconda3/envs/rlena/lib/python3.7/site-packages/numpy/core/fromnumeric.py&quot;, line 85, in _wrapreduction
      return reduction(axis=axis, out=out, **passkwargs)
  TypeError: max() received an invalid combination of arguments - got (out=NoneType, axis=NoneType, ), but expected one of:
   * ()
   * (Tensor other)
   * (int dim, bool keepdim)
        didn't match because some of the keywords were incorrect: out, axis
   * (name dim, bool keepdim)
        didn't match because some of the keywords were incorrect: out, axis\
  Process finished with exit code 1
  
  </code></pre>
  <p>But after I tried that, I'm getting the following TypeError of invalid argument combinations, where it seems like I'm inputting NoneType for both arguments. I have no idea where I made a mistake. Would really appreciate if anyone could take a look at my code!</p>
  <p>This is the code for the agent.py</p>
  <pre><code>class DQNAgent(BaseAgent):
      &quot;&quot;&quot;
      DQN from scratch
      &quot;&quot;&quot;
      def __init__(self, env, args, character=characters.Bomber):
          super(DQNAgent, self).__init__(character)
          self.obs_n = env.observation_space.shape[0]  # output_dim
          self.action_n = env.action_space.n  # input_dim
  
          self.epsilon = args.epsilon
          self.eps_decay = args.eps_decay
          self.min_eps = args.min_eps
          self.gamma = args.gamma
          self.lr = args.lr
  
          self.episodes = args.episodes
          self.maxsteps = args.maxsteps
          self.showevery = args.showevery
  
          self.capacity = args.capacity
          self.batch = args.batch
          self.buffer = ReplayBuffer(self.capacity, self.batch)
  
          self.model = nn.Sequential(
              nn.Linear(self.obs_n, 128),
              nn.ReLU(),
              nn.Linear(128, 256),
              nn.ReLU(),
              nn.Linear(256, self.action_n)
          )
  
          self.optim = optim.Adam(self.model.parameters(), self.lr)
          self.MSE_loss = nn.MSELoss()
  
      def forward(self, state):
          state = torch.FloatTensor(state)
          qvals = self.model(state)
          # qvals (20,2,6) state (20,2,201)
          return qvals
  
      def act(self, obs):
          return np.argmax(self.forward(obs))
  
      def update(self, gamma, batch_size):
          batch = self.buffer.sample(batch_size)
  
          states, actions, rewards, next_states, done = batch
          # actions = {list: 20}(batchsize) of list:2
          currQ = self.forward(states)
          expectQ = rewards + gamma * np.max(self.forward(next_states))
          loss = self.MSE_loss(currQ, expectQ) # TODO: try Huber Loss later too
  
          self.optim.zero_grad()
          loss.backward()
          self.optim.step()
  
      def epsdecay(self):
          self.epsilon = self.epsilon * self.eps_decay if self.epsilon &gt; self.min_eps else self.epsilon
  
  </code></pre>
  <p>And this is the main.py</p>
  <pre><code>def main():
      parser = argparse.ArgumentParser(description='DQN pommerman MARL')
      parser.add_argument('--episodes', type=int, default=3000, help='episodes')
      parser.add_argument('--maxsteps', type=int, default=200, help='maximum steps')
      parser.add_argument('--showevery', type=int, default=300, help='report loss every n episodes')
  
      parser.add_argument('--epsilon', type=float, default=0.05, help='parameter for epsilon greedy')
      parser.add_argument('--eps_decay', type=float, default=0.995, help='epsilon decay rate')
      parser.add_argument('--min_eps', type=float, default=0.05, help='minimum epsilon for decaying')
      parser.add_argument('--gamma', type=float, default=0.95, help='gamma')
      parser.add_argument('--lr', type=float, default=0.01, help='learning rate')
  
      parser.add_argument('--capacity', type=int, default=100000, help='capacity for replay buffer')
      parser.add_argument('--batch', type=int, default=201, help='batch size for replay buffer')
  
      parser.add_argument('--gpu', type=str, default='0', help='gpu number')
  
      args = parser.parse_args()
  
      # GPU
      args.device = torch.device('cuda:{}'.format(args.gpu) if torch.cuda.is_available() else &quot;cpu&quot;)
      print(&quot;GPU using status: &quot;, args.device)
  
      agent_list = [agents.SimpleAgent(), agents.SimpleAgent()]  # placeholder
      env = pommerman.make('OneVsOne-v0', agent_list)
  
      agent1 = DQNAgent(env, args)  # TODO: assertionerror; not agents.BaseAgent??
      agent2 = agents.SimpleAgent()
  
      agent_list = [agent1, agent2]
      # env = pommerman.make('OneVsOne-v0', agent_list)
  
      episode_rewards = []
      action_n = env.action_space.n
  
      for episode in range(args.episodes):
          states = env.reset()
          state_feature = featurize(env, states)
          done = False
          episode_reward = 0
          for step in range(args.maxsteps):
              # if agent1.epsilon &gt; random.random():
              #     action = random.randint(0, action_n - 1)
              # else:
              #     action = agent1.act(state_feature[0])
  
              env.render()
  
              actions = env.act(states)
              # TODO: env.set_training_agent(agents[-1].agent_id)으로 training agent를 명시
              # 그리고 위에 env.act(states)에서 training agent action만 따로 append하기
  
              next_state, reward, done, info = env.step(actions)  # n-array with action for each agent
              next_state_feature = featurize(env, next_state)
              episode_reward += (reward[0]+reward[1])
              agent1.buffer.append([state_feature, actions, reward, next_state_feature, done])
              # env.get_observation -&gt; 찾아보기 (forward model)
  
              if len(agent1.buffer) &gt; args.batch:
                  agent1.update(args.gamma, args.batch)
  
          if done:
              episode_rewards.append(episode_reward)
          if episode % args.showevery == 0:
              print(f&quot;Episode: {episode + 1:2d} finished, result: {'Win' if 0 in info.get('winners', []) else 'Lose'}&quot;)
              print(f&quot;Avg Episode Reward: {np.mean(episode_rewards)}&quot;)
  
          agent1.epsdecay()
  
      env.close()
  </code></pre>