How does one dynamically add new parameters to optimizers in Pytorch?
I was going through this post in the pytorch forum, and I also wanted to do this. The original post removes and adds layers but I think my situation is not that different. I also want to add layers or more filters or word embeddings. My main motivation is that the AI agent does not know the whole vocabulary/dictionary in advance because its large. I prefer strongly (for the moment) to not do character by character RNNs.

So what will happen for me is when the agent starts a forward pass it might find new words it has never seen and will need to add them to the embedding table (or perhaps add new filters before it starts the forward pass).

So what I want to make sure is:


embeddings are added correctly (at the right time, when a new computation graph is made) so that they are updatable by the optimizer
no issues with stored info of past parameters e.g. if its using some sort of momentum


How does one do this? Any sample code that works?

Just to add an answer to the title of your question: "How does one dynamically add new parameters to optimizers in Pytorch?"

You can append params at any time to the optimizer:

import torch
import torch.optim as optim

model = torch.nn.Linear(2, 2) 

# Initialize optimizer
optimizer = optim.Adam(model.parameters(), lr=0.001, momentum=0.9)

extra_params = torch.randn(2, 2)
optimizer.param_groups.append({'params': extra_params })

#then you can print your `extra_params`
print("extra params", extra_params)
print("optimizer params", optimizer.param_groups)

