Removing Categorial Features in Linear Regression
I think using Linear Regression is not a good option as, 


This performs very well on numeric variables(categorical -> binary).
Cannot handle Missing Data(suggestible to ignore those records).
When you are trying to predict, By chance a category is not trained then it cannot predict. 
For example: A variable has 4 categories and in the random sample it
could pick only 3 categories, if the test has 4th category then it
will throw an error. So, we should be careful when the data is
divided between Test and Train


Now, what are the other algorithms which are available:


Random Forest(you cannot use this when any category variable has more than 52 categories, in your case it shouldn't be an issue)
XGBoost


Do let me know if you need any additional explanations.