Would this work to prevent forgetting: train a neural net with N nodes. Then, add more nodes and stop training the original nodes
Per Neil's line of questioning:


If you're doing offline, supervised or reinforcement learning, where you have the dataset and decide what to show to the network next, there's no reason to do this. It's better to just adjust your training to show rare examples to the network more often.
If you're doing online reinforcement learning, then the agent typically controls which examples it sees next through its choice of actions. Increasing the agent's exploration rate will cause it to see a more diverse set of examples, but these examples are also less likely to be useful for solving the task the agent is working on.
If you're doing supervised, online, learning, and you don't control the order in which examples appear, it might be useful to freeze some of the weights, but it might be better, and would certainly be simpler, to increase the learning rate for rare classes instead (in effect, showing the network a given rare example repeatedly when it does show up, so that it takes longer to forget it).