nltk sentence tokenizer gives AttributeError
Question: I am very new to python and NLTK. One issue is baffling me:

When I do

    
    
    tokenized = custom_sent_tokenizer.tokenize("some long text")
    

It gives me perfect result. But when I change this hard coded string to a
variable containing huge text, it gives me the error mentioned in subject viz:

    
    
    tokenized = custom_sent_tokenizer.tokenize(text)
    ...
    AttributeError: 'list' object has no attribute 'abbrev_types'
    

Below is my full code:

    
    
    from __future__ import division
    import urllib.request
    import csv
    import nltk
    from string import punctuation
    from nltk.corpus import stopwords
    from nltk.tokenize import PunktSentenceTokenizer
    
    comments = open("CNPS_Comments.txt").read()
    comments_list = comments.split('\n')
    
    custom_sent_tokenizer = PunktSentenceTokenizer(comments_list[:300])
    ##tokenized = custom_sent_tokenizer.tokenize("some long text")
    text=""
    for comment in comments_list:
       text += comment
    
    tokenized = custom_sent_tokenizer.tokenize(text)
    def process_content():
      try:
        for i in tokenized[:5]:
            words = nltk.word_tokenize(i)
            tagged = nltk.pos_tag(words)
            print(tagged)
    
    except Exception as e:
        print(str(e))
    
    
    process_content()
    

I started with python today and there could be many things I am not doing
effectively here.


Answer: The line that's giving you trouble is correct: That's how you're supposed to
use the sentence tokenizer, with a single string as its argument. You're
getting an error because you have created a monster :-)

The Punkt sentence tokenizer is based on an unsupervised algorithm: You give
it a long text, and it figures out where the sentence boundaries must lie. But
you have trained your tokenizer with a _list_ of sentences (the first 300
elements in `comments_list`), which is incorrect. Somehow the tokenizer
doesn't notice, and gives you something that errors out when you try to use it
properly.

To fix the problem, train your tokenizer with a single string. You can best
join a list of strings into one like this:

    
    
    tokenizer = PunktSentenceTokenizer(" ".join(comments_list[:300]))
    

PS. You must be wrong about it working successfully when you tokenized a
literal string. Certainly there were other differences between the code that
worked, and the code in your question.

