NLP sentiment analysis in Norwegian
All of this:

def get_tokens_for_model(cleaned_tokens_list):
    final_token_list = [] 
    for token in cleaned_tokens_list:
        token_dict = {}
        token_dict.update({token : True})
        final_token_list.append(token_dict)
    return final_token_list

positive_tokens_for_model = get_tokens_for_model(positive_lexicon)
negative_tokens_for_model = get_tokens_for_model(negative_lexicon)

positive_dataset = [(token, "Positive")
                     for token in positive_tokens_for_model]

negative_dataset = [(token, "Negative")
                     for token in negative_tokens_for_model]

dataset = positive_dataset + negative_dataset
#shuffle dataset
for i in range(len(dataset)-1, 0, -1): 
    j = random.randint(0, i + 1)  
    dataset[i], dataset[j] = dataset[j], dataset[i]  


Is just a very verbose way to write:

dataset = [({token : True}, "Positive") for token in positive_lexicon]
dataset.extend([({token : True}, "Negative") for token in negative_lexicon])
random.shuffle(dataset)


The shuffling probabilities might be a bit different, though. random.shuffle basically has for each element equal probability to end up at any index. I think your method has a bias, but I'm not quite sure.

At least your custom shuffling seems to be unbiased:



However, I had to fix it to this, because otherwise it raises an IndexError, because random.randint is inclusive of the end (in contrast to range, slices and random.randrange):

#shuffle dataset
for i in range(len(dataset)-1, 0, -1): 
    j = random.randint(0, i)  
    dataset[i], dataset[j] = dataset[j], dataset[i]