What is the best way to split a sentence for a keyword extraction task?
I think the tagging approach has some merit here. The frequency drop you're observing as a consequence of using this is to be expected, I think. After all, keywords are the words that help differentiate a document from others in a corpus. If you have access to an ontology related to the subject matter of your corpus, you could try to map your rare keyword tags to the ontology, and use parent-level information for each to get a slightly more general set of keyword tags! If this is an approach that interests you, Stanford's open source protégé system is a good framework for working on this.