is there an adequate number of levels of categorical variables? I have a project that I'm working on. The dataset contains many categorical variables and some of them have too many levels (+100). My question is : is there any advice to know the &quot;adequate&quot; number of levels of a variable ? is it based on the number of levels of other variables ? (for example most variables have between 10 and 30 levels and one or two variables have 80 100 levels).
For the variables that contain too many levels, I want to take 80% of most frequent levels and put the 20% into a new level &quot;others&quot; but I don't know at which number of levels I should stop (for example :
var 1 : 70 levels, var 2 : 100 level, var 3 : 13, var 4 : 30, var 5 : 60, should I apply the 80-20 method starting from 60? 70? 100?)
I don't know if I'm being clear but I hope you understand
No, there's no &quot;adequate&quot; number of levels for a categorical variable.
The choice to simplify the data by discarding some levels (for example by using a default category, as you propose) depends on what the goal is (and also number of instances, etc.). Very often this choice is made experimentally, that is by trying different methods (e.g. different thresholds) and observing which one gives the best performance: here you could do a program which tries the different proportions as threshold, then train and test the resulting model for every value, and finally plot the performance for every value.