How to derive weight and bias in a neural network?
This is the Network:

    class Net(torch.nn.Module):
        def __init__(self, n_feature, n_hidden, n_output):  
            super(Net, self).__init__()  
            self.hidden = torch.nn.Linear(n_feature, n_hidden)  
            self.predict = torch.nn.Linear(n_hidden, n_output)   # output layer  
        def forward(self, x): 
            x = F.relu(self.hidden(x))      # activation function for hidden layer 
            x = self.predict(x)             # linear output
            return x

    net = Net(n_feature=1, n_hidden=10, n_output=1)   
    pytorch_total_params = sum(p.numel() for p in net.parameters())
    print(pytorch_total_params)

    w = list(net.parameters())
    print(w)




This is the runnung result:
31
[Parameter containing:
tensor([[ 0.9534],
        [-0.0309],
        [-0.9570],
        [-0.4179],
        [-0.3757],
        [-0.4227],
        [-0.8866],
        [ 0.2107],
        [ 0.0222],
        [ 0.2531]], requires_grad=True), Parameter containing:
tensor([-0.0358, -0.2533,  0.2979,  0.9777,  0.9606,  0.9460,  0.9059,  0.7582,
        -0.5286,  0.3367], requires_grad=True), Parameter containing:
tensor([[-0.2863, -0.3157,  0.2086, -0.0011, -0.0415, -0.2574, -0.0683, -0.0788,
         -0.0339, -0.0195]], requires_grad=True), Parameter containing:
tensor([0.2031], requires_grad=True)]


I don't know why is the number of parameters be 31?
And also don't understand the numbers printed above.(whether it is weight or bias

Because I thought in Relu function, there will only be (2 parameters*10) which is weight and bias multiply 10 hidden layers.

If you print the named parameters you can see to which layer a parameter belongs.
Printing the named parameters:

for p in net.named_parameters():
    print(p)


Creates the following output:

('hidden.weight', Parameter containing:
tensor([[ 0.8324],
        [ 0.2166],
        [-0.9786],
        [ 0.3977],
        [ 0.9008],
        [-0.3102],
        [ 0.5052],
        [ 0.6589],
        [ 0.0828],
        [ 0.6505]], requires_grad=True))
('hidden.bias', Parameter containing:
tensor([ 0.6715,  0.5503, -0.6043,  0.1102, -0.2700,  0.7203, -0.6524, -0.6332,
        -0.2513, -0.1316], requires_grad=True))
('predict.weight', Parameter containing:
tensor([[ 0.1486,  0.1528, -0.0835, -0.3050,  0.1184, -0.0422, -0.2786, -0.2549,
         -0.1532, -0.0255]], requires_grad=True))
('predict.bias', Parameter containing:
tensor([0.2878], requires_grad=True))


As you can see the layers are connected by 10 weights each, as you expected, but there is one bias per neuron on the right side of a 'connection'.
So you have 10 bias-parameters between your input and your hidden layer and just one for the calculation of your final prediction.

You are calculating the input to each neuron in the l-th layer like this weighted sum:



So you need a weight for every connection between the neurons of the two layers, but only one bias per neuron in the l-th layer.

In your case:


input to hidden: 10 weights and 10 bias, because your hidden layer has 10 neurons
hidden to output/predict: 10 weights and 1 bias, because you output a single value


sums up to 31 parameters.


