how to generate sample dataset for classification problem
I know this isn't answering the question that you actually asked, but I suggest that you NOT generate data for your 'short text' categorization problem.

Generated data can work for certain cases when data scientists who are very familiar with an algorithm want to demonstrate a specific feature, but there is a hokeyness that may lead you astray as someone new to data science and machine learning.

I suggest you avail yourself of some free and open data here, or an entire stack exchange dedicated to open data.

Real data will give you experience with less contrived problems and you can often try out similar ML models on very different data sets in order to gain experience with the variations and pitfalls of data science. Be creative... you might not find exactly what you want, but you can subset the data by projecting out a few text fields and using them to classify another field.

Addendum update:

With your recent edit, it is now apparent that you are seeking to turn your unsupervised training data into supervised training data in order to train a supervised learning classification model.  The method that you suggested, "classify some dataset using regex or lucene rule based matches and manually verify them", is a deterministic unsupervised method without the human verification step. 

Without the human verification of the target data that you have created, I would not consider feeding the derived targets back into the classification algortihm as your results will only be as strong as the derived target data and predictions will show similar errors. Instead, you should think about a semi-supervised learning method where you perhaps employ a clustering algorithm and then label the clusters with the target variable.

With human verification of the target data that you have created, this will work just fine as training data for a classification model. The only issue is that this can become tedious. There are mechanical turks (humans paid to perform repetitive tasks) that you can hire to perhaps perform the labeling for you, which may be a more scalable option.

Hope this helps!