Handling categorical missing values ML
First of all I would look how many missing values there are in the column. If there are too much (~20%, generally its difficult to say how much is too much), I would drop the column because imputing 20 % of your data (without prior expert knowledge) or even more probably does not give you meaningful information anymore.  

Secondly I would look at correlations between missing values and other features. Maybe you are lucky and there is some correlation between missing values in column x and a categorical value in column y. Simply look at conditional distributions.

If you choose to impute, check the distribution of the categorical values for non-missing entries. If the distribution is heavily skewed, say there is 95% value 0 and only 5% value 1, you can use the median to impute. Again, the question is how informative this is in the end. Otherwise create an additional categorical value which simply represents a missing value.