NoisyNet DQN with default parameters not exploring I implemented a DQN algorithm that plays OpenAIs Cartpole environment. The NN architecture consists of 3 normal linear layers that encode the state, and one noisy linear layer, that predicts the Q value based on the encoded state.
My NoisyLinear layers looks like this:

class NoisyLinear(nn.Module):
  def __init__(self, in_features, out_features):
    super(NoisyLinear, self).__init__()
    self.in_features = in_features
    self.out_features = out_features
    self.sigma_zero = 0.5
    self.weight_mu = torch.empty(out_features, in_features)
    self.weight_sigma = torch.empty(out_features, in_features)
    self.weight_epsilon = torch.empty(out_features, in_features, requires_grad=False)
    self.bias_mu = torch.empty(out_features)
    self.bias_sigma = torch.empty(out_features)
    self.bias_epsilon = torch.empty(out_features, requires_grad=False)
    self.reset_parameters()
    self.reset_noise()

  def reset_parameters(self):
    mu_range = 1 / math.sqrt(self.in_features)
    self.weight_mu.data.uniform_(-mu_range, mu_range)
    self.weight_sigma.data.fill_(self.sigma_zero / math.sqrt(self.in_features))
    self.bias_mu.data.uniform_(-mu_range, mu_range)
    self.bias_sigma.data.fill_(self.sigma_zero / math.sqrt(self.out_features))

  def _scale_noise(self, size):
    x = torch.randn(size)
    return x.sign().mul_(x.abs().sqrt_())

  def reset_noise(self):
    epsilon_in = self._scale_noise(self.in_features)
    epsilon_out = self._scale_noise(self.out_features)
    self.weight_epsilon.copy_(epsilon_out.ger(epsilon_in))
    self.bias_epsilon.copy_(epsilon_out)

  def forward(self, input):
    return F.linear(input, self.weight_mu + self.weight_sigma * self.weight_epsilon, self.bias_mu + self.bias_sigma * self.bias_epsilon)


However, with the default hyperparameters from the (sigma_0 = 0.5), the agent does not explore at all, and even if I crank it up to sigma_0 = 5, it works way worse than epsilon greedy.
(When I use noisy nets I don't use epsilon greedy).
I think the problem is that you are not defining the weights and biases as parameters. So, when you backpropagate, they are not modified.

These lines should do the trick:

self.weight_mu = Parameter(torch.Tensor(out_features, in_features))
self.weight_sigma = Parameter(torch.Tensor(out_features, in_features))
self.bias_mu = Parameter(torch.Tensor(out_features))
self.bias_sigma = Parameter(torch.Tensor(out_features))


In case you are not familiar, the Parameter class must be imported from torch:

from torch.nn.parameter import Parameter