Prevent model from over-focusing on strong features I have a classification model (DNN/Linear layers with some transformers and other things later). The input to the model are several different modalities of different lengths and different amounts of information. I am trying to mitigate the dimensionality difference by projecting different modalities into the same dimensional space and then combining them into the same space. However, the model seems to be focusing completely on the strongest input features and almost completely ignores the abundance of weaker ones. All of the input features are normalized (bools to 0/1 and continuous to mean=0, stddev=1), so the issue is not the scale of feature values but the predictive power of those few features which end up choking others.
Are there any methods out there for addressing this?
Use dropout.
Dropout is a form of regularization where you zero out the input components with a certain probability.
If you want your model to learn representations that rely less on some features, just place a dropout layer before those features enter the rest of the network. This will force the network to learn more robust representations that depend less on the strongest features.