Checking if english sentences have impact catpured in them using NLP
The approach you propose is ok but it can be improved in my opinion:


Find a corpus (or several corpora), chosen to be representative of the kind data you expect to process eventually. Ideally it would be directly a sample of the target data. 
Label all the sentences in the corpus or a random subset. This is because you need to preserve the distribution as much as possible, especially the proportion of positive/negative instances. If you start from the set of filtered sentences you'll have two problems: 


a proportion of positive cases higher than in regular text, and this is likely to make your model over-predict positive cases;
sentences containing only the selected trigger words, and this will cause your model to predict as negative any sentence which doesn't have a trigger word (that's a problem since your list of trigger words can't be exhaustive).



Ideally you would manually label all your corpus, but that's probably not realistic. That's why you could try to use your idea of trigger words in a slightly different way in order to label all the instances efficiently:


Filter the sentences which contain any trigger word and label these; this is your initial training set
Use a semi-supervised approach to label the rest of the sentences (maybe you could consider active learning).