a neural network can be used as an optimizer? I was wondering if a neural network can be used as an optimizer.
In other words, a network2 used as an optimizer takes the loss value of network1 and based on that it predicts the best weights for network 1.
as a discriminator in architecture, gan imitates a function of loss.
Theoretically yes, pratically no.
just think About what you Need for this second Network. You Need a dataset X of real numbers representing loss and you Need y representing weight plus bias.
You can already see that weights of a slightly different Architecture will be different. So creating such dataset is just infeasable given that you Need a lot of datapoints for your Network to learn.