Is it possible to use different L1 / L2 regularization parameters for different sets of weights in chainer or pytorch?
(As an example) When implementing a simple linear model for noutput target values as a neural network in pytorch: 

l1=L.Linear(ninput, noutput)


(call)

y = self.l1(x)
return y


Adding this hook will do L2 regularization on all weights, imposing the same alpha=0.01 everywhere:

optimizer.add_hook(optimizer.WeightDecay(rate=0.01))


Is it possible to use a different alpha for each set of weights leading from all ninput input units to one of the noutput output units?

Since we are working in pytorch it is possible to add other scalars to loss function yourself. So assume loss from you classfier is L ( assume it is a cross entropy loss ) and you have a linear layer defined as:

l1 = nn.Linear(in,out)


Now if you want to have different regularization on each set of weights then all you have to do is gather weights using ( i.e select using index) and add to the final loss:

loss = L (crossentropy loss) + sum ( alpha * norm(l1.weight[k]))


alpha the hyper-parameters and norm is mostly L2 norm,in pytorch it is just torch.norm(l1.weight) where index k is a tensor of indices of weights you want to select. Finally, you don't need to do the global regularization as you have done in the code.
