What is the most effective way to build a classifier?
Your approach would definitely work. I would recommend training a variety of classifiers and comparing their performance using multiclass roc analysis. Also, think about other useful features in addition to the ones you mentioned (e.g. pos tag). Feature engineering is one of the most important factors in building good predictive models. Another thing to keep in mind is that the classes could be highly imbalanced which might influence your model's performance.