Is there any problem with dropping only part of the OneHot generated features? The one hot encoder adds more columns to the data, one for each category in the encoded feature. In the example below, the column City was transformed into 4 other columns.
Suppose a Decision Tree is ran on a dataset the below is part of and City_Chicago and City_New_York appear to be in top most important features while City_Detroit and City_SanFrancisco in the least important. Would there be any problem if I drop City_Detroit and City_SanFrancisco from my dataset, but keep City_Chicago and City_New_York or do I need to keep all city features as they are part of one initial feature?
|---------------------|------------------|-------------|---------------|---------------|
|      City           | City_SanFrancisco| City_Detroit| City_New_York | City_Chicago  |
|---------------------|------------------|-------------|---------------|---------------|
|     San Francisco   |         1        |      0      |       0       |     0         |
|---------------------|------------------|-------------|---------------|---------------|
|     Detroit         |         0        |      1      |       0       |     0         |
|---------------------|------------------|-------------|---------------|---------------|
|     New York        |         0        |      0      |       1       |     0         |
|---------------------|------------------|-------------|---------------|---------------|
|     Chicago         |         0        |      0      |       0       |     1         |
|---------------------|------------------|-------------|---------------|---------------|
I think you can keep as many as you want, and it'll be alright. Sometimes it's even worth to delete the very rare classes to have more stable features.
In addition, for linear regression, you shouldn't include all of them, as you might have a collinearity issue.
To sum up, no problem with not keeping them all.