Why does a simpler model performs better than a complicated one? This has happened to me, a complicated model couldn't solve the problem when a simpler one solved it in a few epochs. How is that? I believed that a more complicated model means more number of parameters and more number of parameters means a higher capability to solve a problem.
I have heard that people say, A simpler Model can perform well because it has a low variance. But, what does that even mean?
High variance means that your model's performance varies from data to data, which is bad for the model.
For example:
You used a polynomial classifier with high degree, so it will overfit your training data resulting in good accuracy, but when tried on a different dataset it will yield less accuracy , resulting in variance.


The image on the right is way too complex which results in overfitting.

The middle image shows a simpler model than the image on right, but fits the data better ie. generalize the data better