1. A computer-implemented method for a machine learning system that mitigates catastrophic forgetting, comprising: receiving a first input at a first node in a first layer of a hierarchy of nodes, wherein the first input comprises a) a first previous feature vector that was generated by the first node based on a first previous input and b) a data item; processing at least a portion of the first input by the first node to generate a first feature vector, the first feature vector comprising a first plurality of feature elements; processing a second input by a second node in a second layer of the hierarchy of nodes to generate a second feature vector, wherein the second input comprises a) at least a portion of the first feature vector and b) a second previous feature vector that was generated by the second node based on a second previous input, and wherein the second feature vector comprises a second plurality of feature elements; generating at least one of a first sparse feature vector from the first feature vector or a second sparse feature vector from the second feature vector, wherein a majority of feature elements in the first sparse feature vector and the second sparse feature vector have a value of zero; and processing at least one of the first sparse feature vector or the second sparse feature vector by a third node to determine a first output.