Cleaning a certain feature to predict salary using Machine Learning Info:
I am working on a dataset, and i would like to create a model that would predict salary.
Columns are as follows:
Index(['ID', 'Salary', 'DOJ', 'DOL', 'Designation', 'JobCity', 'Gender', 'DOB',
       '10percentage', '10board', '12percentage', '12board', 'CollegeTier',
       'Degree', 'Specialization', 'collegeGPA', 'CollegeCityTier',
       'CollegeState', 'English', 'Logical', 'Quant', 'Domain',
       'ComputerProgramming', 'ElectronicsAndSemicon', 'ComputerScience',
       'MechanicalEngg', 'ElectricalEngg', 'TelecomEngg', 'CivilEngg',
       'conscientiousness', 'agreeableness', 'extraversion', 'nueroticism',
       'openess_to_experience'],
      dtype='object')

Issue:
The train dataset contains around 3000 entries. For the feature 'JobCity', there are around 200 unique values. Out of 3000 entries, 180 are cities whose value counts are below 10.
For example, the city 'Agra' occurs only twice and 'Ranchi' occurs only once.
I plan to encode all the cities to feed into the machine learning (polynomial regression) model.
How should the cities with value counts less than 10 be handled?
There are two ways to handle this:

Aggregating values and feature engineering

Instead of simply inputting the individual city, you could try to aggregate this value into a new variable &quot;JobCitySize&quot; to cluster cities by inhabitant size or a variable &quot;Region&quot; to cluster by geographic region. This will increase cases and also help solve the question of how to deal with unseen data.

Defining NA or unseen values

With any model and categorical encoding you have to think ahead what you want to do with &quot;unseen data&quot; in your case cities that aren't present in your training data set. Do you want to set them as NA, give a standard value (&quot;Other&quot;), etc.? Whatever process you use for these values you could also use for any value in your data set that falls below a threshold of &quot;usefulness&quot;.