Creating Rules for Word Permutations Based on a word's POS Tag
Question: I have another question and since this community has been so great helping me
along I thought I would give it another shot.

Right now I have Python 3 code that imports a CSV file where the first column
is full of words in the following format:

    
    
    The
    Words
    Look
    Like
    This
    In
    A
    Column
    

Once this CSV file is uploaded and read by Python, the words are tagged using
an NLTK POS Tagger. From there, permutations are made from all of the words
and then the results are exported to a new CSV file. Right now, my full code
goes like this

    
    
    Import CSV
    with open(r'C:\Users\jkk\Desktop\python.csv', 'r') as f:
        reader = csv.reader(f)
        J = []
        for row in reader:
          J.extend(row)
    import nltk
    D = nltk.pos_tag(J)
    C = list(itertools.permutations(D, 3))
    with open('test.csv', 'w') as a_file:
        for result in C:
        result = ' '.join(result)
        a_file.write(result + '\n')
    

**My question is, how does one make rules for the word permutations based on
the word tag?** More specifically, the reason I am tagging words is because I
don't want nonsense permutations (i.e. The This In / A This In / etc). Once
the words are tagged with their respective part of speech, how do I code rules
based on their tag (for example): Never have two "DT" labeled words follow
each other (i.e. "The" and "A"). Or always have a NN tagged word be followed
by a VBG tagged word (i.e. "Looks" always comes after "Words")? And then
finally, once those rules are implemented, get rid of the tags so that just
the original words remain? I realize this is a general question but any
guidance would be very much appreciated on how to approach this question as I
am still very new and learning every step of the way! Any resources, code, or
even advice would help! Thank you again for taking the time to read this long
post!


Answer: The set of rules that define legal strings in a language is called a grammar
(or formal grammar). There are many formalisms which allow you to express
these rules. One that is reasonnably simple to experiment with is a context
free grammar ([CFG](http://en.wikipedia.org/wiki/Context_free_grammar)). NLTK
comes with tools to generate strings from these. Here is the [NLTK book's
chapter on syntax](http://www.nltk.org/book/ch08.html). They go into much more
depth.

The following code is for python 3 with NLTK 3.0a4. The API changed between
NLTK 2 and 3, so it will not run on the older version.

    
    
    from nltk import ContextFreeGrammar
    from nltk.parse.generate import generate
    from ntlk.util import trigrams
    
    # build a simple grammar
    cfg = """
    S -> NP VP
    VP -> VBZ NP
    NP -> DT | NN | DT NN | DT JJ NN | JJ NN
    """
    
    # you get these from your csv
    words = "this is a simple sentence".split()
    tagged = set(pos_tag(words))
    # Add the words to the grammar
    for word, tag in tagged:
        cfg += "{tag} -> '{word}'\n".format(word=word, tag=tag)
    grammar = parse_cfg(cfg)
    
    valid_trigrams = set()
    
    language = generate(grammar)
    for valid_sentence in language:
        valid_trigrams.update(list(trigrams(valid_sentence)))
    
    print(valid_trigrams)
    # {('simple', 'sentence', 'is'), ('this', 'is', 'this'), ('a', 'sentence', 'is'), ('sentence', 'is', 'a'), ('a', 'is', 'a'), ('this', 'is', 'simple'), ('sentence', 'is', 'this'), ('this', 'is', 'sentence'), ('is', 'a', 'sentence'), ('is', 'a', 'simple'), ('a', 'simple', 'sentence'), ('a', 'is', 'this'), ('this', 'simple', 'sentence'), ('this', 'is', 'a'), ('is', 'simple', 'sentence'), ('a', 'is', 'simple'), ('this', 'sentence', 'is'), ('is', 'this', 'sentence'), ('sentence', 'is', 'sentence'), ('sentence', 'is', 'simple'), ('is', 'this', 'simple'), ('a', 'is', 'sentence')}
    

There is a limitation to this approach though, since a context free grammar
cannot cover all of English. There is no known way of validating syntax for
English anyways though, so you can only have an approximate solution.

Another thing you should be aware of is that the POS tagger assumes that the
order of words is relevant. Roughly, it gives each word a set of possible
tags, then refines it based on the preceding and or following words, so if
you're CSV contains sentences, then you are ok, otherwise, you might want to
specify the unigram pos tagger `nltk.tag.UnigramTagger`, but regardless, you
will only get the most common tag. This will be an issue for words like "run"
which can be a verb or a noun ("a morning run" vs "I run").

