Using NLTK's universalt tagset with non-English corpora
Question: I'm using NLTK (3.0.4-1) in Python 3.4.3+ and I'd like to proccess some of the
tagged corpora using the universal tagset (which I had to install), [as
explained in NLTK book, chapter 5](http://www.nltk.org/book/ch05.html).

I can access any of these corpora with their original PoS tagset, e.g.:

    
    
    from nltk.corpus import brown, cess_esp, floresta
    
    print(brown.tagged_sents()[0])
    [('The', 'AT'), ('Fulton', 'NP-TL'), ('County', 'NN-TL'), ('Grand', 'JJ-TL'), ('Jury', 'NN-TL'), ('said', 'VBD'), ('Friday', 'NR'), ('an', 'AT'), ('investigation', 'NN'), ('of', 'IN'), ("Atlanta's", 'NP$'), ('recent', 'JJ'), ('primary', 'NN'), ('election', 'NN'), ('produced', 'VBD'), ('``', '``'), ('no', 'AT'), ('evidence', 'NN'), ("''", "''"), ('that', 'CS'), ('any', 'DTI'), ('irregularities', 'NNS'), ('took', 'VBD'), ('place', 'NN'), ('.', '.')]
    
    print(cess_esp.tagged_sents()[0])
    [('El', 'da0ms0'), ('grupo', 'ncms000'), ('estatal', 'aq0cs0'), ('Electricité_de_France', 'np00000'), ('-Fpa-', 'Fpa'), ('EDF', 'np00000'), ('-Fpt-', 'Fpt'), ('anunció', 'vmis3s0'), ('hoy', 'rg'), (',', 'Fc'), ('jueves', 'W'), (',', 'Fc'), ('la', 'da0fs0'), ('compra', 'ncfs000'), ('del', 'spcms'), ('51_por_ciento', 'Zp'), ('de', 'sps00'), ('la', 'da0fs0'), ('empresa', 'ncfs000'), ('mexicana', 'aq0fs0'), ('Electricidad_Águila_de_Altamira', 'np00000'), ('-Fpa-', 'Fpa'), ('EAA', 'np00000'), ('-Fpt-', 'Fpt'), (',', 'Fc'), ('creada', 'aq0fsp'), ('por', 'sps00'), ('el', 'da0ms0'), ('japonés', 'aq0ms0'), ('Mitsubishi_Corporation', 'np00000'), ('para', 'sps00'), ('poner_en_marcha', 'vmn0000'), ('una', 'di0fs0'), ('central', 'ncfs000'), ('de', 'sps00'), ('gas', 'ncms000'), ('de', 'sps00'), ('495', 'Z'), ('megavatios', 'ncmp000'), ('.', 'Fp')]
    
    print(floresta.tagged_sents()[0])
    [('Um', '>N+art'), ('revivalismo', 'H+n'), ('refrescante', 'N<+adj')]
    

So far so good, but when I use the option `tagset='universal'` to access the
simplified version of the PoS tags, it works only for the Brown corpus.

    
    
    print(brown.tagged_sents(tagset='universal')[0])
    [('The', 'DET'), ('Fulton', 'NOUN'), ('County', 'NOUN'), ('Grand', 'ADJ'), ('Jury', 'NOUN'), ('said', 'VERB'), ('Friday', 'NOUN'), ('an', 'DET'), ('investigation', 'NOUN'), ('of', 'ADP'), ("Atlanta's", 'NOUN'), ('recent', 'ADJ'), ('primary', 'NOUN'), ('election', 'NOUN'), ('produced', 'VERB'), ('``', '.'), ('no', 'DET'), ('evidence', 'NOUN'), ("''", '.'), ('that', 'ADP'), ('any', 'DET'), ('irregularities', 'NOUN'), ('took', 'VERB'), ('place', 'NOUN'), ('.', '.')]
    

When accessing the corpora in Spanish and Portuguese I get a long chain of
errors and a `LookupError` exception.

    
    
    print(cess_esp.tagged_sents(tagset='universal')[0])
    ---------------------------------------------------------------------------
    LookupError                               Traceback (most recent call last)
    <ipython-input-6-4e2e43e54e2d> in <module>()
    ----> 1 print(cess_esp.tagged_sents(tagset='universal')[0])
    
    [...]
    
    LookupError: 
    **********************************************************************
      Resource 'taggers/universal_tagset/unknown.map' not found.
      Please use the NLTK Downloader to obtain the resource:  >>>
      nltk.download()
      Searched in:
        - '/home/victor/nltk_data'
        - '/usr/share/nltk_data'
        - '/usr/local/share/nltk_data'
        - '/usr/lib/nltk_data'
        - '/usr/local/lib/nltk_data'
        - ''
    **********************************************************************
    

Among the mappings located in my `taggers/universal_tagset` directory, I can
find the mappings for Spanish (`es-cast3lb.map`) and Portuguese (`pt-
bosque.map`), but I don't have any `unknown.map` file. Any ideas how to solve
it?

Thanks in advance :-)


Answer: That's an interesting question. The NLTK implements mapping to the Universal
tagset only for a fixed collection of corpora, with the help of the fixed maps
you found in `nltk_data/taggers/universal_tagset/`. Except for a few special
cases (which include treating the brown corpus as if it was named `en-brown`),
the rule is to look for a mapping file that has the same name as the tagset
used for your corpus. In your case, the tagset is set to "unknown", which is
why you see that message.

Now, are you sure "the mapping for Spanish", i.e. the map `es-cast3lb.map`,
actually matches the tagset for your corpus? I certainly wouldn't just assume
it does, since any project can create their own tagset and rules for use. **If
this is the same tagset your corpus uses,** your problem has an easy solution:

  * When you initialize your corpus reader, e.g. `cess_esp`, add the option `tagset="es-cast3lb"` to the constructor. If necessary, e.g. for corpora already loaded by the NLTK with `tagset="unknown"`, you can override the tagset after initialization like this: 
    
        cess_esp._tagset = "es-cast3lb"
    

This tells the corpus reader what tagset is used in the corpus. After that,
specifying `tagset="universal"` should cause the selected mapping to be
applied.

If **this tagset is not actually suited** to your corpus, your first job is to
study the documentation of the tagset for your corpus, and create an
appropriate mapping to the Universal tagset; as you've probably seen, the
format is pretty trivial. You can then put your mapping in operation by
dropping it in `nltk_data/taggers/universal_tagset`. Adding your own resources
to the `nltk_data` area is decidedly a hack, but if you get this far, I
recommend you contribute your tagset map to the nltk... which will resolve the
hack after the fact.

**Edit:** So (per the comments) it's the right tagset, but only the 1-2 letter
POS tags are in the mapping dictionary (the rest of the tag presumably
describes the features of inflected words). Here's a quick way to extend the
mapping dictionary on the fly, so that you can see the universal tags:

    
    
    import nltk
    from nltk.corpus import cess_esp
    cess_esp._tagset = "es-cast3lb"
    
    nltk.tag.mapping._load_universal_map("es-cast3lb")  # initialize; normally loaded on demand
    mapdict = nltk.tag.mapping._MAPPINGS["es-cast3lb"]["universal"] # shortcut to the map
    
    alltags = set(t for w, t in cess_esp.tagged_words())
    for tag in alltags:
        if len(tag) <= 2:   # These are complete
            continue
        mapdict[tag] = mapdict[tag[:2]]
    

This discards the agreement information. If you'd rather decorate the
"universal" tags with it, just set `mapdict[tag]` to
`mapdict[tag[:2]]+"-"+tag[2:]`.

I'd save this dictionary to a file as described above, so that you don't have
to recompute the mapping every time you load your corpus.

