NLP sentiment analysis in Norwegian Please keep in mind that I am very new to data science and completely new to NLP! I am trying to create a model to classify customer reviews as either negative or positive. However, my main problem is that most of them are in Norwegian, which seems not to be very well supported. I have found this repo https://github.com/ltgoslo/norsentlex/tree/master/Fullform containing both negative and positive lexicons and have decided to use it.

with open("Fullform_Positive_lexicon.txt") as f:
    reader = csv.reader(f)
    positive_lexicon = []
    for row in reader:
        print(str(row))
        positive_lexicon.append(row[0])

with open("Fullform_Negative_lexicon.txt") as f:
    reader = csv.reader(f)
    negative_lexicon = []
    for row in reader:
        negative_lexicon.append(row[0])

#adjusted to use it with NaiveBayesClassifier
def get_tokens_for_model(cleaned_tokens_list):
    final_token_list = [] 
    for token in cleaned_tokens_list:
        token_dict = {}
        token_dict.update({token : True})
        final_token_list.append(token_dict)
    return final_token_list

positive_tokens_for_model = get_tokens_for_model(positive_lexicon)
negative_tokens_for_model = get_tokens_for_model(negative_lexicon)

positive_dataset = [(token, "Positive")
                     for token in positive_tokens_for_model]

negative_dataset = [(token, "Negative")
                     for token in negative_tokens_for_model]

dataset = positive_dataset + negative_dataset
#shuffle dataset
for i in range(len(dataset)-1, 0, -1): 
    j = random.randint(0, i + 1)  
    dataset[i], dataset[j] = dataset[j], dataset[i]  

train_data = dataset[:20000]
test_data = dataset[7742:]

classifier = NaiveBayesClassifier.train(train_data)

print("Accuracy is:", classify.accuracy(classifier, test_data))


So I am just training my model based on this lexicon and then I am trying to apply it to my comments. I am getting here 87%, which is not so bad. However, it looks worse when used with whole sentences.

with open("stopwords.csv") as f:
    reader = csv.reader(f)
    norwegian_stopwords = []
    for row in reader:
        norwegian_stopwords.append(row[0])

customer_feedback = pd.read_excel("classification_sample.xlsx")
customer_feedback = customer_feedback.dropna()
customer_feedback['Kommentar'] = customer_feedback['Kommentar'].apply(remove_punctuation)

list_of_comments = list(customer_feedback['Kommentar'])

for comment in list_of_comments:
    custom_tokens = word_tokenize(comment)
    filtered_tokens = []
    for word in custom_tokens:
        if word not in  norwegian_stopwords:
            filtered_tokens.append(word)

    classification = classifier.classify(dict([token, True] for token in filtered_tokens))
    probdist = classifier.prob_classify(dict([token, True] for token in filtered_tokens))
    pred_sentiment = probdist.max()
    print("Sentence: " + comment)
    print("Classified as: " + str(classification))
    print("Key words: " + str(custom_tokens))
    print("Probability: " + str(round(probdist.prob(pred_sentiment), 2)))
    print("-----------------------------------------------------------")


I know my code is not the highest quality right now(although suggestions regarding this are also appreciated). I am looking mostly for some feedback what more can I do to improve my accuracy. What is currently not very clear to me, is how to properly train the model on words and then achieve the very same accuracy on sentences.
All of this:

def get_tokens_for_model(cleaned_tokens_list):
    final_token_list = [] 
    for token in cleaned_tokens_list:
        token_dict = {}
        token_dict.update({token : True})
        final_token_list.append(token_dict)
    return final_token_list

positive_tokens_for_model = get_tokens_for_model(positive_lexicon)
negative_tokens_for_model = get_tokens_for_model(negative_lexicon)

positive_dataset = [(token, "Positive")
                     for token in positive_tokens_for_model]

negative_dataset = [(token, "Negative")
                     for token in negative_tokens_for_model]

dataset = positive_dataset + negative_dataset
#shuffle dataset
for i in range(len(dataset)-1, 0, -1): 
    j = random.randint(0, i + 1)  
    dataset[i], dataset[j] = dataset[j], dataset[i]  


Is just a very verbose way to write:

dataset = [({token : True}, "Positive") for token in positive_lexicon]
dataset.extend([({token : True}, "Negative") for token in negative_lexicon])
random.shuffle(dataset)


The shuffling probabilities might be a bit different, though. random.shuffle basically has for each element equal probability to end up at any index. I think your method has a bias, but I'm not quite sure.

At least your custom shuffling seems to be unbiased:



However, I had to fix it to this, because otherwise it raises an IndexError, because random.randint is inclusive of the end (in contrast to range, slices and random.randrange):

#shuffle dataset
for i in range(len(dataset)-1, 0, -1): 
    j = random.randint(0, i)  
    dataset[i], dataset[j] = dataset[j], dataset[i]