Initial commit

.gitignore

@@ -0,0 +1,108 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+env/
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*,cover
+.hypothesis/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# IPython Notebook
+.ipynb_checkpoints
+
+# pyenv
+.python-version
+
+# celery beat schedule file
+celerybeat-schedule
+
+# dotenv
+.env
+
+# virtualenv
+venv/
+ENV/
+
+# Spyder project settings
+.spyderproject
+
+# Rope project settings
+.ropeproject
+
+# Local data
+/data/local
+
+# Vim swapfiles
+*.swp
+*.swo
+
+# nosetests
+.noseids
+
+# pyTorch model
+pytorch_model.bin
+
+# VSCODE
+.vscode/*
+
+# data
+*.csv

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2017 Bjarke Felbo, Han Thi Nguyen, Thomas Wolf
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,82 @@
+# torchMoji
+
+TorchMoji is a [pyTorch](http://pytorch.org/) implementation of the [DeepMoji](https://github.com/bfelbo/DeepMoji) model developped by Bjarke Felbo, Alan Mislove, Anders Søgaard, Iyad Rahwan and Sune Lehmann.
+
+This model trained on 1.2 billion tweets with emojis to understand how language is used to express emotions. Through transfer learning the model can obtain state-of-the-art performance on many emotion-related text modeling tasks.
+  
+Try the online demo of DeepMoji [http://deepmoji.mit.edu](http://deepmoji.mit.edu/)! See the [paper](https://arxiv.org/abs/1708.00524), [blog post](https://medium.com/@bjarkefelbo/what-can-we-learn-from-emojis-6beb165a5ea0) or [FAQ](https://www.media.mit.edu/projects/deepmoji/overview/) for more details.
+
+## Overview
+* [torchmoji/](torchmoji) contains all the underlying code needed to convert a dataset to the vocabulary and use the model.
+* [examples/](examples) contains short code snippets showing how to convert a dataset to the vocabulary, load up the model and run it on that dataset.
+* [scripts/](scripts) contains code for processing and analysing datasets to reproduce results in the paper.
+* [model/](model) contains the pretrained model and vocabulary.
+* [data/](data) contains raw and processed datasets that we include in this repository for testing.
+* [tests/](tests) contains unit tests for the codebase.
+  
+To start out with, have a look inside the [examples/](examples) directory. See [score_texts_emojis.py](examples/score_texts_emojis.py) for how to use DeepMoji to extract emoji predictions, [encode_texts.py](examples/encode_texts.py) for how to convert text into 2304-dimensional emotional feature vectors or [finetune_youtube_last.py](examples/finetune_youtube_last.py) for how to use the model for transfer learning on a new dataset.
+
+Please consider citing the [paper](https://arxiv.org/abs/1708.00524) of DeepMoji if you use the model or code (see below for citation).
+
+## Installation
+
+We assume that you're using [Python 2.7-3.5](https://www.python.org/downloads/) with [pip](https://pip.pypa.io/en/stable/installing/) installed.
+
+First you need to install [pyTorch (version 0.2+)](http://pytorch.org/), currently by:
+```bash
+conda install pytorch -c soumith
+```
+At the present stage the model can't make efficient use of CUDA. See details in the HuggingFace blog post.
+
+When pyTorch is installed, run the following in the root directory to install the remaining dependencies:
+  
+```bash
+pip install -e .
+```
+This will install the following dependencies:
+* [scikit-learn](https://github.com/scikit-learn/scikit-learn)
+* [text-unidecode](https://github.com/kmike/text-unidecode)
+* [emoji](https://github.com/carpedm20/emoji)
+
+Then, run the download script to downloads the pretrained torchMoji weights (~85MB) from [here](https://www.dropbox.com/s/q8lax9ary32c7t9/pytorch_model.bin?dl=0) and put them in the model/ directory:
+
+```bash
+python scripts/download_weights.py
+```
+
+## Testing
+To run the tests, install [nose](http://nose.readthedocs.io/en/latest/). After installing, navigate to the [tests/](tests) directory and run:
+
+```bash
+cd tests
+nosetests -v
+```
+
+By default, this will also run finetuning tests. These tests train the model for one epoch and then check the resulting accuracy, which may take several minutes to finish. If you'd prefer to exclude those, run the following instead: 
+
+```bash
+cd tests
+nosetests -v -a '!slow'
+```
+
+## Disclaimer 
+This code has been tested to work with Python 2.7 and 3.5 on Ubuntu 16.04 and macOS Sierra machines. It has not been optimized for efficiency, but should be fast enough for most purposes. We do not give any guarantees that there are no bugs - use the code on your own responsibility!
+
+## Contributions
+We welcome pull requests if you feel like something could be improved. You can also greatly help us by telling us how you felt when writing your most recent tweets. Just click [here](http://deepmoji.mit.edu/contribute/) to contribute.
+
+## License
+This code and the pretrained model is licensed under the MIT license. 
+
+## Benchmark datasets
+The benchmark datasets are uploaded to this repository for convenience purposes only. They were not released by us and we do not claim any rights on them. Use the datasets at your responsibility and make sure you fulfill the licenses that they were released with. If you use any of the benchmark datasets please consider citing the original authors.
+
+## Citation
+```
+@inproceedings{felbo2017,
+  title={Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm},
+  author={Felbo, Bjarke and Mislove, Alan and S{\o}gaard, Anders and Rahwan, Iyad and Lehmann, Sune},
+  booktitle={Conference on Empirical Methods in Natural Language Processing (EMNLP)},
+  year={2017}
+}
+```

data/.gitkeep

@@ -0,0 +1 @@
+ 

data/filtering/wanted_emojis.csv

@@ -0,0 +1,64 @@
+\U0001f602
+\U0001f612
+\U0001f629
+\U0001f62d
+\U0001f60d
+\U0001f614
+\U0001f44c
+\U0001f60a
+\u2764
+\U0001f60f
+\U0001f601
+\U0001f3b6
+\U0001f633
+\U0001f4af
+\U0001f634
+\U0001f60c
+\u263a
+\U0001f64c
+\U0001f495
+\U0001f611
+\U0001f605
+\U0001f64f
+\U0001f615
+\U0001f618
+\u2665
+\U0001f610
+\U0001f481
+\U0001f61e
+\U0001f648
+\U0001f62b
+\u270c
+\U0001f60e
+\U0001f621
+\U0001f44d
+\U0001f622
+\U0001f62a
+\U0001f60b
+\U0001f624
+\u270b
+\U0001f637
+\U0001f44f
+\U0001f440
+\U0001f52b
+\U0001f623
+\U0001f608
+\U0001f613
+\U0001f494
+\u2661
+\U0001f3a7
+\U0001f64a
+\U0001f609
+\U0001f480
+\U0001f616
+\U0001f604
+\U0001f61c
+\U0001f620
+\U0001f645
+\U0001f4aa
+\U0001f44a
+\U0001f49c
+\U0001f496
+\U0001f499
+\U0001f62c
+\u2728

examples/.gitkeep

@@ -0,0 +1 @@
+ 

examples/README.md

@@ -0,0 +1,31 @@
+# torchMoji examples
+
+## Initialization  
+[create_twitter_vocab.py](create_twitter_vocab.py)  
+Create a new vocabulary from a tsv file.  
+  
+[tokenize_dataset.py](tokenize_dataset.py)  
+Tokenize a given dataset using the prebuilt vocabulary.  
+  
+[vocab_extension.py](vocab_extension.py)  
+Extend the given vocabulary using dataset-specific words.  
+  
+[dataset_split.py](dataset_split.py)  
+Split a given dataset into training, validation and testing.
+  
+## Use pretrained model/architecture 
+[score_texts_emojis.py](score_texts_emojis.py)  
+Use torchMoji to score texts for emoji distribution.  
+
+[encode_texts.py](encode_texts.py)  
+Use torchMoji to encode the text into 2304-dimensional feature vectors for further modeling/analysis.
+
+## Transfer learning
+[finetune_youtube_last.py](finetune_youtube_last.py)  
+Finetune the model on the SS-Youtube dataset using the 'last' method.  
+    
+[finetune_insults_chain-thaw.py](finetune_insults_chain-thaw.py)  
+Finetune the model on the Kaggle insults dataset (from blog post) using the 'chain-thaw' method.  
+  
+[finetune_semeval_class-avg_f1.py](finetune_semeval_class-avg_f1.py)  
+Finetune the model on the SemeEval emotion dataset using the 'full' method and evaluate using the class average F1 metric.  

examples/create_twitter_vocab.py

@@ -0,0 +1,13 @@
+""" Creates a vocabulary from a tsv file.
+"""
+
+import codecs
+import example_helper
+from torchmoji.create_vocab import VocabBuilder
+from torchmoji.word_generator import TweetWordGenerator
+
+with codecs.open('../../twitterdata/tweets.2016-09-01', 'rU', 'utf-8') as stream:
+    wg = TweetWordGenerator(stream)
+    vb = VocabBuilder(wg)
+    vb.count_all_words()
+    vb.save_vocab()

examples/dataset_split.py

@@ -0,0 +1,59 @@
+'''
+Split a given dataset into three different datasets: training, validation and
+testing.
+
+This is achieved by splitting the given list of sentences into three separate
+lists according to either a given ratio (e.g. [0.7, 0.1, 0.2]) or by an
+explicit enumeration. The sentences are also tokenised using the given
+vocabulary.
+
+Also splits a given list of dictionaries containing information about
+each sentence.
+
+An additional parameter can be set 'extend_with', which will extend the given
+vocabulary with up to 'extend_with' tokens, taken from the training dataset.
+'''
+from __future__ import print_function, unicode_literals
+import example_helper
+import json
+
+from torchmoji.sentence_tokenizer import SentenceTokenizer
+
+DATASET = [
+    'I am sentence 0',
+    'I am sentence 1',
+    'I am sentence 2',
+    'I am sentence 3',
+    'I am sentence 4',
+    'I am sentence 5',
+    'I am sentence 6',
+    'I am sentence 7',
+    'I am sentence 8',
+    'I am sentence 9 newword',
+    ]
+
+INFO_DICTS = [
+    {'label': 'sentence 0'},
+    {'label': 'sentence 1'},
+    {'label': 'sentence 2'},
+    {'label': 'sentence 3'},
+    {'label': 'sentence 4'},
+    {'label': 'sentence 5'},
+    {'label': 'sentence 6'},
+    {'label': 'sentence 7'},
+    {'label': 'sentence 8'},
+    {'label': 'sentence 9'},
+    ]
+
+with open('../model/vocabulary.json', 'r') as f:
+    vocab = json.load(f)
+st = SentenceTokenizer(vocab, 30)
+
+# Split using the default split ratio
+print(st.split_train_val_test(DATASET, INFO_DICTS))
+
+# Split explicitly
+print(st.split_train_val_test(DATASET,
+                              INFO_DICTS,
+                              [[0, 1, 2, 4, 9], [5, 6], [7, 8, 3]],
+                              extend_with=1))

examples/encode_texts.py

@@ -0,0 +1,41 @@
+# -*- coding: utf-8 -*-
+
+""" Use torchMoji to encode texts into emotional feature vectors.
+"""
+from __future__ import print_function, division, unicode_literals
+import json
+
+from torchmoji.sentence_tokenizer import SentenceTokenizer
+from torchmoji.model_def import torchmoji_feature_encoding
+from torchmoji.global_variables import PRETRAINED_PATH, VOCAB_PATH
+
+TEST_SENTENCES = ['I love mom\'s cooking',
+                  'I love how you never reply back..',
+                  'I love cruising with my homies',
+                  'I love messing with yo mind!!',
+                  'I love you and now you\'re just gone..',
+                  'This is shit',
+                  'This is the shit']
+
+maxlen = 30
+batch_size = 32
+
+print('Tokenizing using dictionary from {}'.format(VOCAB_PATH))
+with open(VOCAB_PATH, 'r') as f:
+    vocabulary = json.load(f)
+st = SentenceTokenizer(vocabulary, maxlen)
+tokenized, _, _ = st.tokenize_sentences(TEST_SENTENCES)
+
+print('Loading model from {}.'.format(PRETRAINED_PATH))
+model = torchmoji_feature_encoding(PRETRAINED_PATH)
+print(model)
+
+print('Encoding texts..')
+encoding = model(tokenized)
+
+print('First 5 dimensions for sentence: {}'.format(TEST_SENTENCES[0]))
+print(encoding[0,:5])
+
+# Now you could visualize the encodings to see differences,
+# run a logistic regression classifier on top,
+# or basically anything you'd like to do.

examples/example_helper.py

@@ -0,0 +1,6 @@
+""" Module import helper.
+Modifies PATH in order to allow us to import the torchmoji directory.
+"""
+import sys
+from os.path import abspath, dirname
+sys.path.insert(0, dirname(dirname(abspath(__file__))))

examples/finetune_insults_chain-thaw.py

@@ -0,0 +1,44 @@
+"""Finetuning example.
+
+Trains the torchMoji model on the kaggle insults dataset, using the 'chain-thaw'
+finetuning method and the accuracy metric. See the blog post at
+https://medium.com/@bjarkefelbo/what-can-we-learn-from-emojis-6beb165a5ea0
+for more information. Note that results may differ a bit due to slight
+changes in preprocessing and train/val/test split.
+
+The 'chain-thaw' method does the following:
+0) Load all weights except for the softmax layer. Extend the embedding layer if
+   necessary, initialising the new weights with random values.
+1) Freeze every layer except the last (softmax) layer and train it.
+2) Freeze every layer except the first layer and train it.
+3) Freeze every layer except the second etc., until the second last layer.
+4) Unfreeze all layers and train entire model.
+"""
+
+from __future__ import print_function
+import example_helper
+import json
+from torchmoji.model_def import torchmoji_transfer
+from torchmoji.global_variables import PRETRAINED_PATH
+from torchmoji.finetuning import (
+     load_benchmark,
+     finetune)
+
+
+DATASET_PATH = '../data/kaggle-insults/raw.pickle'
+nb_classes = 2
+
+with open('../model/vocabulary.json', 'r') as f:
+    vocab = json.load(f)
+
+# Load dataset. Extend the existing vocabulary with up to 10000 tokens from
+# the training dataset.
+data = load_benchmark(DATASET_PATH, vocab, extend_with=10000)
+
+# Set up model and finetune. Note that we have to extend the embedding layer
+# with the number of tokens added to the vocabulary.
+model = torchmoji_transfer(nb_classes, PRETRAINED_PATH, extend_embedding=data['added'])
+print(model)
+model, acc = finetune(model, data['texts'], data['labels'], nb_classes,
+                      data['batch_size'], method='chain-thaw')
+print('Acc: {}'.format(acc))

examples/finetune_semeval_class-avg_f1.py

@@ -0,0 +1,50 @@
+"""Finetuning example.
+
+Trains the torchMoji model on the SemEval emotion dataset, using the 'last'
+finetuning method and the class average F1 metric.
+
+The 'last' method does the following:
+0) Load all weights except for the softmax layer. Do not add tokens to the
+   vocabulary and do not extend the embedding layer.
+1) Freeze all layers except for the softmax layer.
+2) Train.
+
+The class average F1 metric does the following:
+1) For each class, relabel the dataset into binary classification
+   (belongs to/does not belong to this class).
+2) Calculate F1 score for each class.
+3) Compute the average of all F1 scores.
+"""
+
+from __future__ import print_function
+import example_helper
+import json
+from torchmoji.finetuning import load_benchmark
+from torchmoji.class_avg_finetuning import class_avg_finetune
+from torchmoji.model_def import torchmoji_transfer
+from torchmoji.global_variables import PRETRAINED_PATH
+
+DATASET_PATH = '../data/SE0714/raw.pickle'
+nb_classes = 3
+
+with open('../model/vocabulary.json', 'r') as f:
+    vocab = json.load(f)
+
+
+# Load dataset. Extend the existing vocabulary with up to 10000 tokens from
+# the training dataset.
+data = load_benchmark(DATASET_PATH, vocab, extend_with=10000)
+
+# Set up model and finetune. Note that we have to extend the embedding layer
+# with the number of tokens added to the vocabulary.
+#
+# Also note that when using class average F1 to evaluate, the model has to be
+# defined with two classes, since the model will be trained for each class
+# separately.
+model = torchmoji_transfer(2, PRETRAINED_PATH, extend_embedding=data['added'])
+print(model)
+
+# For finetuning however, pass in the actual number of classes.
+model, f1 = class_avg_finetune(model, data['texts'], data['labels'],
+                                nb_classes, data['batch_size'], method='last')
+print('F1: {}'.format(f1))

examples/finetune_youtube_last.py

@@ -0,0 +1,35 @@
+"""Finetuning example.
+
+Trains the torchMoji model on the SS-Youtube dataset, using the 'last'
+finetuning method and the accuracy metric.
+
+The 'last' method does the following:
+0) Load all weights except for the softmax layer. Do not add tokens to the
+   vocabulary and do not extend the embedding layer.
+1) Freeze all layers except for the softmax layer.
+2) Train.
+"""
+
+from __future__ import print_function
+import example_helper
+import json
+from torchmoji.model_def import torchmoji_transfer
+from torchmoji.global_variables import PRETRAINED_PATH, VOCAB_PATH, ROOT_PATH
+from torchmoji.finetuning import (
+     load_benchmark,
+     finetune)
+
+DATASET_PATH = '{}/data/SS-Youtube/raw.pickle'.format(ROOT_PATH)
+nb_classes = 2
+
+with open(VOCAB_PATH, 'r') as f:
+    vocab = json.load(f)
+
+# Load dataset.
+data = load_benchmark(DATASET_PATH, vocab)
+
+# Set up model and finetune
+model = torchmoji_transfer(nb_classes, PRETRAINED_PATH)
+print(model)
+model, acc = finetune(model, data['texts'], data['labels'], nb_classes, data['batch_size'], method='last')
+print('Acc: {}'.format(acc))

examples/score_texts_emojis.py

@@ -0,0 +1,76 @@
+# -*- coding: utf-8 -*-
+
+""" Use torchMoji to score texts for emoji distribution.
+
+The resulting emoji ids (0-63) correspond to the mapping
+in emoji_overview.png file at the root of the torchMoji repo.
+
+Writes the result to a csv file.
+"""
+from __future__ import print_function, division, unicode_literals
+import example_helper
+import json
+import csv
+import numpy as np
+
+from torchmoji.sentence_tokenizer import SentenceTokenizer
+from torchmoji.model_def import torchmoji_emojis
+from torchmoji.global_variables import PRETRAINED_PATH, VOCAB_PATH
+
+OUTPUT_PATH = 'test_sentences.csv'
+
+TEST_SENTENCES = ['I love mom\'s cooking',
+                  'I love how you never reply back..',
+                  'I love cruising with my homies',
+                  'I love messing with yo mind!!',
+                  'I love you and now you\'re just gone..',
+                  'This is shit',
+                  'This is the shit']
+
+
+def top_elements(array, k):
+    ind = np.argpartition(array, -k)[-k:]
+    return ind[np.argsort(array[ind])][::-1]
+
+maxlen = 30
+
+print('Tokenizing using dictionary from {}'.format(VOCAB_PATH))
+with open(VOCAB_PATH, 'r') as f:
+    vocabulary = json.load(f)
+
+st = SentenceTokenizer(vocabulary, maxlen)
+
+print('Loading model from {}.'.format(PRETRAINED_PATH))
+model = torchmoji_emojis(PRETRAINED_PATH)
+print(model)
+print('Running predictions.')
+tokenized, _, _ = st.tokenize_sentences(TEST_SENTENCES)
+prob = model(tokenized)
+
+for prob in [prob]:
+    # Find top emojis for each sentence. Emoji ids (0-63)
+    # correspond to the mapping in emoji_overview.png 
+    # at the root of the torchMoji repo.
+    print('Writing results to {}'.format(OUTPUT_PATH))
+    scores = []
+    for i, t in enumerate(TEST_SENTENCES):
+        t_tokens = tokenized[i]
+        t_score = [t]
+        t_prob = prob[i]
+        ind_top = top_elements(t_prob, 5)
+        t_score.append(sum(t_prob[ind_top]))
+        t_score.extend(ind_top)
+        t_score.extend([t_prob[ind] for ind in ind_top])
+        scores.append(t_score)
+        print(t_score)
+
+    with open(OUTPUT_PATH, 'wb') as csvfile:
+        writer = csv.writer(csvfile, delimiter=',', lineterminator='\n')
+        writer.writerow(['Text', 'Top5%',
+                        'Emoji_1', 'Emoji_2', 'Emoji_3', 'Emoji_4', 'Emoji_5',
+                        'Pct_1', 'Pct_2', 'Pct_3', 'Pct_4', 'Pct_5'])
+        for i, row in enumerate(scores):
+            try:
+                writer.writerow(row)
+            except:
+                print("Exception at row {}!".format(i))

examples/tokenize_dataset.py

@@ -0,0 +1,26 @@
+"""
+Take a given list of sentences and turn it into a numpy array, where each
+number corresponds to a word. Padding is used (number 0) to ensure fixed length
+of sentences.
+"""
+
+from __future__ import print_function, unicode_literals
+import example_helper
+import json
+from torchmoji.sentence_tokenizer import SentenceTokenizer
+
+with open('../model/vocabulary.json', 'r') as f:
+    vocabulary = json.load(f)
+
+st = SentenceTokenizer(vocabulary, 30)
+test_sentences = [
+    '\u2014 -- \u203c !!\U0001F602',
+    'Hello world!',
+    'This is a sample tweet #example',
+    ]
+
+tokens, infos, stats = st.tokenize_sentences(test_sentences)
+
+print(tokens)
+print(infos)
+print(stats)

examples/vocab_extension.py

@@ -0,0 +1,30 @@
+"""
+Extend the given vocabulary using dataset-specific words.
+
+1. First create a vocabulary for the specific dataset.
+2. Find all words not in our vocabulary, but in the dataset vocabulary.
+3. Take top X (default=1000) of these words and add them to the vocabulary.
+4. Save this combined vocabulary and embedding matrix, which can now be used.
+"""
+
+from __future__ import print_function, unicode_literals
+import example_helper
+import json
+from torchmoji.create_vocab import extend_vocab, VocabBuilder
+from torchmoji.word_generator import WordGenerator
+
+new_words = ['#zzzzaaazzz', 'newword', 'newword']
+word_gen = WordGenerator(new_words)
+vb = VocabBuilder(word_gen)
+vb.count_all_words()
+
+with open('../model/vocabulary.json') as f:
+    vocab = json.load(f)
+
+print(len(vocab))
+print(vb.word_counts)
+extend_vocab(vocab, vb, max_tokens=1)
+
+# 'newword' should be added because it's more frequent in the given vocab
+print(vocab['newword'])
+print(len(vocab))

model/.gitkeep

@@ -0,0 +1 @@
+ 

scripts/analyze_all_results.py

@@ -0,0 +1,40 @@
+from __future__ import print_function
+
+# allow us to import the codebase directory
+import sys
+import glob
+import numpy as np
+from os.path import dirname, abspath
+sys.path.insert(0, dirname(dirname(abspath(__file__))))
+
+DATASETS = ['SE0714', 'Olympic', 'PsychExp', 'SS-Twitter', 'SS-Youtube',
+            'SCv1', 'SV2-GEN'] # 'SE1604' excluded due to Twitter's ToS
+
+def get_results(dset):
+    METHOD = 'last'
+    RESULTS_DIR = 'results/'
+    RESULT_PATHS = glob.glob('{}/{}_{}_*_results.txt'.format(RESULTS_DIR, dset, METHOD))
+    assert len(RESULT_PATHS)
+
+    scores = []
+    for path in RESULT_PATHS:
+        with open(path) as f:
+            score = f.readline().split(':')[1]
+        scores.append(float(score))
+
+    average = np.mean(scores)
+    maximum = max(scores)
+    minimum = min(scores)
+    std = np.std(scores)
+
+    print('Dataset: {}'.format(dset))
+    print('Method:  {}'.format(METHOD))
+    print('Number of results: {}'.format(len(scores)))
+    print('--------------------------')
+    print('Average: {}'.format(average))
+    print('Maximum: {}'.format(maximum))
+    print('Minimum: {}'.format(minimum))
+    print('Standard deviaton: {}'.format(std))
+
+for dset in DATASETS:
+    get_results(dset)

scripts/analyze_results.py

@@ -0,0 +1,39 @@
+from __future__ import print_function
+
+import sys
+import glob
+import numpy as np
+
+DATASET = 'SS-Twitter' # 'SE1604' excluded due to Twitter's ToS
+METHOD = 'new'
+
+# Optional usage: analyze_results.py <dataset> <method>
+if len(sys.argv) == 3:
+    DATASET = sys.argv[1]
+    METHOD = sys.argv[2]
+
+RESULTS_DIR = 'results/'
+RESULT_PATHS = glob.glob('{}/{}_{}_*_results.txt'.format(RESULTS_DIR, DATASET, METHOD))
+
+if not RESULT_PATHS:
+    print('Could not find results for \'{}\' using \'{}\' in directory \'{}\'.'.format(DATASET, METHOD, RESULTS_DIR))
+else:
+    scores = []
+    for path in RESULT_PATHS:
+        with open(path) as f:
+            score = f.readline().split(':')[1]
+        scores.append(float(score))
+
+    average = np.mean(scores)
+    maximum = max(scores)
+    minimum = min(scores)
+    std = np.std(scores)
+
+    print('Dataset: {}'.format(DATASET))
+    print('Method:  {}'.format(METHOD))
+    print('Number of results: {}'.format(len(scores)))
+    print('--------------------------')
+    print('Average: {}'.format(average))
+    print('Maximum: {}'.format(maximum))
+    print('Minimum: {}'.format(minimum))
+    print('Standard deviaton: {}'.format(std))

scripts/calculate_coverages.py

@@ -0,0 +1,85 @@
+from __future__ import print_function
+import pickle
+import json
+import csv
+import sys
+from io import open
+
+# Allow us to import the torchmoji directory
+from os.path import dirname, abspath
+sys.path.insert(0, dirname(dirname(abspath(__file__))))
+
+from torchmoji.sentence_tokenizer import SentenceTokenizer, coverage
+
+IS_PYTHON2 = int(sys.version[0]) == 2
+
+OUTPUT_PATH = 'coverage.csv'
+DATASET_PATHS = [
+    '../data/Olympic/raw.pickle',
+    '../data/PsychExp/raw.pickle',
+    '../data/SCv1/raw.pickle',
+    '../data/SCv2-GEN/raw.pickle',
+    '../data/SE0714/raw.pickle',
+    #'../data/SE1604/raw.pickle', # Excluded due to Twitter's ToS
+    '../data/SS-Twitter/raw.pickle',
+    '../data/SS-Youtube/raw.pickle',
+    ]
+
+with open('../model/vocabulary.json', 'r') as f:
+    vocab = json.load(f)
+
+results = []
+for p in DATASET_PATHS:
+    coverage_result = [p]
+    print('Calculating coverage for {}'.format(p))
+    with open(p, 'rb') as f:
+        if IS_PYTHON2:
+            s = pickle.load(f)
+        else:
+            s = pickle.load(f, fix_imports=True)
+
+    # Decode data
+    try:
+        s['texts'] = [unicode(x) for x in s['texts']]
+    except UnicodeDecodeError:
+        s['texts'] = [x.decode('utf-8') for x in s['texts']]
+
+    # Own
+    st = SentenceTokenizer({}, 30)
+    tests, dicts, _ = st.split_train_val_test(s['texts'], s['info'],
+                                              [s['train_ind'],
+                                               s['val_ind'],
+                                               s['test_ind']],
+                                              extend_with=10000)
+    coverage_result.append(coverage(tests[2]))
+
+    # Last
+    st = SentenceTokenizer(vocab, 30)
+    tests, dicts, _ = st.split_train_val_test(s['texts'], s['info'],
+                                              [s['train_ind'],
+                                               s['val_ind'],
+                                               s['test_ind']],
+                                              extend_with=0)
+    coverage_result.append(coverage(tests[2]))
+
+    # Full
+    st = SentenceTokenizer(vocab, 30)
+    tests, dicts, _ = st.split_train_val_test(s['texts'], s['info'],
+                                              [s['train_ind'],
+                                               s['val_ind'],
+                                               s['test_ind']],
+                                              extend_with=10000)
+    coverage_result.append(coverage(tests[2]))
+
+    results.append(coverage_result)
+
+with open(OUTPUT_PATH, 'wb') as csvfile:
+    writer = csv.writer(csvfile, delimiter='\t', lineterminator='\n')
+    writer.writerow(['Dataset', 'Own', 'Last', 'Full'])
+    for i, row in enumerate(results):
+        try:
+            writer.writerow(row)
+        except:
+            print("Exception at row {}!".format(i))
+
+print('Saved to {}'.format(OUTPUT_PATH))

scripts/convert_all_datasets.py

@@ -0,0 +1,105 @@
+from __future__ import print_function
+
+import json
+import math
+import pickle
+import sys
+from io import open
+import numpy as np
+from os.path import abspath, dirname
+sys.path.insert(0, dirname(dirname(abspath(__file__))))
+
+from torchmoji.word_generator import WordGenerator
+from torchmoji.create_vocab import VocabBuilder
+from torchmoji.sentence_tokenizer import SentenceTokenizer, extend_vocab, coverage
+from torchmoji.tokenizer import tokenize
+
+IS_PYTHON2 = int(sys.version[0]) == 2
+
+DATASETS = [
+    'Olympic',
+    'PsychExp',
+    'SCv1',
+    'SCv2-GEN',
+    'SE0714',
+    #'SE1604', # Excluded due to Twitter's ToS
+    'SS-Twitter',
+    'SS-Youtube',
+    ]
+
+DIR = '../data'
+FILENAME_RAW = 'raw.pickle'
+FILENAME_OWN = 'own_vocab.pickle'
+FILENAME_OUR = 'twitter_vocab.pickle'
+FILENAME_COMBINED = 'combined_vocab.pickle'
+
+
+def roundup(x):
+    return int(math.ceil(x / 10.0)) * 10
+
+
+def format_pickle(dset, train_texts, val_texts, test_texts, train_labels, val_labels, test_labels):
+    return {'dataset': dset,
+            'train_texts': train_texts,
+            'val_texts': val_texts,
+            'test_texts': test_texts,
+            'train_labels': train_labels,
+            'val_labels': val_labels,
+            'test_labels': test_labels}
+
+def convert_dataset(filepath, extend_with, vocab):
+    print('-- Generating {} '.format(filepath))
+    sys.stdout.flush()
+    st = SentenceTokenizer(vocab, maxlen)
+    tokenized, dicts, _ = st.split_train_val_test(texts,
+                                                  labels,
+                                                  [data['train_ind'],
+                                                   data['val_ind'],
+                                                   data['test_ind']],
+                                                  extend_with=extend_with)
+    pick = format_pickle(dset, tokenized[0], tokenized[1], tokenized[2],
+                        dicts[0], dicts[1], dicts[2])
+    with open(filepath, 'w') as f:
+        pickle.dump(pick, f)
+    cover = coverage(tokenized[2])
+
+    print('     done. Coverage: {}'.format(cover))
+
+with open('../model/vocabulary.json', 'r') as f:
+    vocab = json.load(f)
+
+for dset in DATASETS:
+    print('Converting {}'.format(dset))
+
+    PATH_RAW = '{}/{}/{}'.format(DIR, dset, FILENAME_RAW)
+    PATH_OWN = '{}/{}/{}'.format(DIR, dset, FILENAME_OWN)
+    PATH_OUR = '{}/{}/{}'.format(DIR, dset, FILENAME_OUR)
+    PATH_COMBINED = '{}/{}/{}'.format(DIR, dset, FILENAME_COMBINED)
+
+    with open(PATH_RAW, 'rb') as dataset:
+        if IS_PYTHON2:
+            data = pickle.load(dataset)
+        else:
+            data = pickle.load(dataset, fix_imports=True)
+
+    # Decode data
+    try:
+        texts = [unicode(x) for x in data['texts']]
+    except UnicodeDecodeError:
+        texts = [x.decode('utf-8') for x in data['texts']]
+
+    wg = WordGenerator(texts)
+    vb = VocabBuilder(wg)
+    vb.count_all_words()
+
+    # Calculate max length of sequences considered
+    # Adjust batch_size accordingly to prevent GPU overflow
+    lengths = [len(tokenize(t)) for t in texts]
+    maxlen = roundup(np.percentile(lengths, 80.0))
+
+    # Extract labels
+    labels = [x['label'] for x in data['info']]
+
+    convert_dataset(PATH_OWN, 50000, {})
+    convert_dataset(PATH_OUR, 0, vocab)
+    convert_dataset(PATH_COMBINED, 10000, vocab)

scripts/download_weights.py

@@ -0,0 +1,64 @@
+from __future__ import print_function
+import os
+from subprocess import call
+
+curr_folder = os.path.basename(os.path.normpath(os.getcwd()))
+
+weights_filename = 'pytorch_model.bin'
+weights_folder = 'model'
+weights_path = '{}/{}'.format(weights_folder, weights_filename)
+if curr_folder == 'scripts':
+    weights_path = '../' + weights_path
+weights_download_link = 'https://www.dropbox.com/s/q8lax9ary32c7t9/pytorch_model.bin?dl=0#'
+
+
+MB_FACTOR = float(1<<20)
+
+def prompt():
+    while True:
+        valid = {
+            'y': True,
+            'ye': True,
+            'yes': True,
+            'n': False,
+            'no': False,
+        }
+        choice = raw_input().lower()
+        if choice in valid:
+            return valid[choice]
+        else:
+            print('Please respond with \'y\' or \'n\' (or \'yes\' or \'no\')')
+
+download = True
+if os.path.exists(weights_path):
+    print('Weight file already exists at {}. Would you like to redownload it anyway? [y/n]'.format(weights_path))
+    download = prompt()
+    already_exists = True
+else:
+    already_exists = False
+
+if download:
+    print('About to download the pretrained weights file from {}'.format(weights_download_link))
+    if already_exists == False:
+        print('The size of the file is roughly 85MB. Continue? [y/n]')
+    else:
+        os.unlink(weights_path)
+
+    if already_exists or prompt():
+        print('Downloading...')
+
+        #urllib.urlretrieve(weights_download_link, weights_path)
+        #with open(weights_path,'wb') as f:
+        #    f.write(requests.get(weights_download_link).content)
+
+        # downloading using wget due to issues with urlretrieve and requests
+        sys_call = 'wget {} -O {}'.format(weights_download_link, os.path.abspath(weights_path))
+        print("Running system call: {}".format(sys_call))
+        call(sys_call, shell=True)
+
+        if os.path.getsize(weights_path) / MB_FACTOR < 80:
+            raise ValueError("Download finished, but the resulting file is too small! " +
+                             "It\'s only {} bytes.".format(os.path.getsize(weights_path)))
+        print('Downloaded weights to {}'.format(weights_path))
+else:
+    print('Exiting.')

scripts/finetune_dataset.py

@@ -0,0 +1,109 @@
+""" Finetuning example.
+"""
+from __future__ import print_function
+import sys
+import numpy as np
+from os.path import abspath, dirname
+sys.path.insert(0, dirname(dirname(abspath(__file__))))
+
+import json
+import math
+from torchmoji.model_def import torchmoji_transfer
+from torchmoji.global_variables import PRETRAINED_PATH, VOCAB_PATH
+from torchmoji.finetuning import (
+     load_benchmark,
+     finetune)
+from torchmoji.class_avg_finetuning import class_avg_finetune
+
+def roundup(x):
+    return int(math.ceil(x / 10.0)) * 10
+
+
+# Format: (dataset_name,
+#          path_to_dataset,
+#          nb_classes,
+#          use_f1_score)
+DATASETS = [
+     #('SE0714', '../data/SE0714/raw.pickle', 3, True),
+     #('Olympic', '../data/Olympic/raw.pickle', 4, True),
+     #('PsychExp', '../data/PsychExp/raw.pickle', 7, True),
+     #('SS-Twitter', '../data/SS-Twitter/raw.pickle', 2, False),
+     ('SS-Youtube', '../data/SS-Youtube/raw.pickle', 2, False),
+     #('SE1604', '../data/SE1604/raw.pickle', 3, False), # Excluded due to Twitter's ToS
+     #('SCv1', '../data/SCv1/raw.pickle', 2, True),
+     #('SCv2-GEN', '../data/SCv2-GEN/raw.pickle', 2, True)
+      ]
+
+RESULTS_DIR = 'results'
+
+# 'new' | 'last' | 'full' | 'chain-thaw'
+FINETUNE_METHOD = 'last'
+VERBOSE = 1
+
+nb_tokens = 50000
+nb_epochs = 1000
+epoch_size = 1000
+
+with open(VOCAB_PATH, 'r') as f:
+    vocab = json.load(f)
+
+for rerun_iter in range(5):
+    for p in DATASETS:
+
+        # debugging
+        assert len(vocab) == nb_tokens
+
+        dset = p[0]
+        path = p[1]
+        nb_classes = p[2]
+        use_f1_score = p[3]
+
+        if FINETUNE_METHOD == 'last':
+            extend_with = 0
+        elif FINETUNE_METHOD in ['new', 'full', 'chain-thaw']:
+            extend_with = 10000
+        else:
+            raise ValueError('Finetuning method not recognised!')
+
+        # Load dataset.
+        data = load_benchmark(path, vocab, extend_with=extend_with)
+
+        (X_train, y_train) = (data['texts'][0], data['labels'][0])
+        (X_val, y_val) = (data['texts'][1], data['labels'][1])
+        (X_test, y_test) = (data['texts'][2], data['labels'][2])
+
+        weight_path = PRETRAINED_PATH if FINETUNE_METHOD != 'new' else None
+        nb_model_classes = 2 if use_f1_score else nb_classes
+        model = torchmoji_transfer(
+                    nb_model_classes,
+                    data['maxlen'], weight_path,
+                    extend_embedding=data['added'])
+        model.summary()
+
+        # Training
+        print('Training: {}'.format(path))
+        if use_f1_score:
+            model, result = class_avg_finetune(model, data['texts'],
+                                               data['labels'],
+                                               nb_classes, data['batch_size'],
+                                               FINETUNE_METHOD,
+                                               verbose=VERBOSE)
+        else:
+            model, result = finetune(model, data['texts'], data['labels'],
+                                     nb_classes, data['batch_size'],
+                                     FINETUNE_METHOD, metric='acc',
+                                     verbose=VERBOSE)
+
+        # Write results
+        if use_f1_score:
+            print('Overall F1 score (dset = {}): {}'.format(dset, result))
+            with open('{}/{}_{}_{}_results.txt'.
+                      format(RESULTS_DIR, dset, FINETUNE_METHOD, rerun_iter),
+                      "w") as f:
+                f.write("F1: {}\n".format(result))
+        else:
+            print('Test accuracy (dset = {}): {}'.format(dset, result))
+            with open('{}/{}_{}_{}_results.txt'.
+                      format(RESULTS_DIR, dset, FINETUNE_METHOD, rerun_iter),
+                      "w") as f:
+                f.write("Acc: {}\n".format(result))

scripts/results/.gitkeep

@@ -0,0 +1 @@
+ 

setup.py

@@ -0,0 +1,16 @@
+from setuptools import setup
+
+setup(
+    name='torchmoji',
+    version='1.0',
+    packages=['torchmoji'],
+    description='torchMoji',
+    include_package_data=True,
+    install_requires=[
+        'emoji==0.4.5',
+        'numpy==1.13.1',
+        'scipy==0.19.1',
+        'scikit-learn==0.19.0',
+        'text-unidecode==1.0',
+    ],
+)

tests/test_finetuning.py

@@ -0,0 +1,235 @@
+from __future__ import absolute_import, print_function, division, unicode_literals
+
+import test_helper
+
+from nose.plugins.attrib import attr
+import json
+import numpy as np
+
+from torchmoji.class_avg_finetuning import relabel
+from torchmoji.sentence_tokenizer import SentenceTokenizer
+
+from torchmoji.finetuning import (
+    calculate_batchsize_maxlen,
+    freeze_layers,
+    change_trainable,
+    finetune,
+    load_benchmark
+    )
+from torchmoji.model_def import (
+    torchmoji_transfer,
+    torchmoji_feature_encoding,
+    torchmoji_emojis
+    )
+from torchmoji.global_variables import (
+    PRETRAINED_PATH,
+    NB_TOKENS,
+    VOCAB_PATH,
+    ROOT_PATH
+    )
+
+
+def test_calculate_batchsize_maxlen():
+    """ Batch size and max length are calculated properly.
+    """
+    texts = ['a b c d',
+             'e f g h i']
+    batch_size, maxlen = calculate_batchsize_maxlen(texts)
+
+    assert batch_size == 250
+    assert maxlen == 10, maxlen
+
+
+def test_freeze_layers():
+    """ Correct layers are frozen.
+    """
+    model = torchmoji_transfer(5)
+    keyword = 'output_layer'
+
+    model = freeze_layers(model, unfrozen_keyword=keyword)
+
+    for name, module in model.named_children():
+        trainable = keyword.lower() in name.lower()
+        assert all(p.requires_grad == trainable for p in module.parameters())
+
+
+def test_change_trainable():
+    """ change_trainable() changes trainability of layers.
+    """
+    model = torchmoji_transfer(5)
+    change_trainable(model.embed, False)
+    assert not any(p.requires_grad for p in model.embed.parameters())
+    change_trainable(model.embed, True)
+    assert all(p.requires_grad for p in model.embed.parameters())
+
+
+def test_torchmoji_transfer_extend_embedding():
+    """ Defining torchmoji with extension.
+    """
+    extend_with = 50
+    model = torchmoji_transfer(5, weight_path=PRETRAINED_PATH,
+                              extend_embedding=extend_with)
+    embedding_layer = model.embed
+    assert embedding_layer.weight.size()[0] == NB_TOKENS + extend_with
+
+
+def test_torchmoji_return_attention():
+    seq_tensor = np.array([[1]])
+    # test the output of the normal model
+    model = torchmoji_emojis(weight_path=PRETRAINED_PATH)
+    # check correct number of outputs
+    assert len(model(seq_tensor)) == 1
+    # repeat above described tests when returning attention weights
+    model = torchmoji_emojis(weight_path=PRETRAINED_PATH, return_attention=True)
+    assert len(model(seq_tensor)) == 2
+
+
+def test_relabel():
+    """ relabel() works with multi-class labels.
+    """
+    nb_classes = 3
+    inputs = np.array([
+        [True, False, False],
+        [False, True, False],
+        [True, False, True],
+    ])
+    expected_0 = np.array([True, False, True])
+    expected_1 = np.array([False, True, False])
+    expected_2 = np.array([False, False, True])
+
+    assert np.array_equal(relabel(inputs, 0, nb_classes), expected_0)
+    assert np.array_equal(relabel(inputs, 1, nb_classes), expected_1)
+    assert np.array_equal(relabel(inputs, 2, nb_classes), expected_2)
+
+
+def test_relabel_binary():
+    """ relabel() works with binary classification (no changes to labels)
+    """
+    nb_classes = 2
+    inputs = np.array([True, False, False])
+
+    assert np.array_equal(relabel(inputs, 0, nb_classes), inputs)
+
+
+@attr('slow')
+def test_finetune_full():
+    """ finetuning using 'full'.
+    """
+    DATASET_PATH = ROOT_PATH+'/data/SS-Youtube/raw.pickle'
+    nb_classes = 2
+    # Keras and pyTorch implementation of the Adam optimizer are slightly different and change a bit the results
+    # We reduce the min accuracy needed here to pass the test
+    # See e.g. https://discuss.pytorch.org/t/suboptimal-convergence-when-compared-with-tensorflow-model/5099/11
+    min_acc = 0.68
+
+    with open(VOCAB_PATH, 'r') as f:
+        vocab = json.load(f)
+
+    data = load_benchmark(DATASET_PATH, vocab, extend_with=10000)
+    print('Loading pyTorch model from {}.'.format(PRETRAINED_PATH))
+    model = torchmoji_transfer(nb_classes, PRETRAINED_PATH, extend_embedding=data['added'])
+    print(model)
+    model, acc = finetune(model, data['texts'], data['labels'], nb_classes,
+                          data['batch_size'], method='full', nb_epochs=1)
+
+    print("Finetune full SS-Youtube 1 epoch acc: {}".format(acc))
+    assert acc >= min_acc
+
+
+@attr('slow')
+def test_finetune_last():
+    """ finetuning using 'last'.
+    """
+    dataset_path = ROOT_PATH + '/data/SS-Youtube/raw.pickle'
+    nb_classes = 2
+    min_acc = 0.68
+
+    with open(VOCAB_PATH, 'r') as f:
+        vocab = json.load(f)
+
+    data = load_benchmark(dataset_path, vocab)
+    print('Loading model from {}.'.format(PRETRAINED_PATH))
+    model = torchmoji_transfer(nb_classes, PRETRAINED_PATH)
+    print(model)
+    model, acc = finetune(model, data['texts'], data['labels'], nb_classes,
+                          data['batch_size'], method='last', nb_epochs=1)
+
+    print("Finetune last SS-Youtube 1 epoch acc: {}".format(acc))
+
+    assert acc >= min_acc
+
+
+def test_score_emoji():
+    """ Emoji predictions make sense.
+    """
+    test_sentences = [
+        'I love mom\'s cooking',
+        'I love how you never reply back..',
+        'I love cruising with my homies',
+        'I love messing with yo mind!!',
+        'I love you and now you\'re just gone..',
+        'This is shit',
+        'This is the shit'
+    ]
+
+    expected = [
+        np.array([36,  4,  8, 16, 47]),
+        np.array([1, 19, 55, 25, 46]),
+        np.array([31,  6, 30, 15, 13]),
+        np.array([54, 44,  9, 50, 49]),
+        np.array([46,  5, 27, 35, 34]),
+        np.array([55, 32, 27,  1, 37]),
+        np.array([48, 11,  6, 31,  9])
+    ]
+
+    def top_elements(array, k):
+        ind = np.argpartition(array, -k)[-k:]
+        return ind[np.argsort(array[ind])][::-1]
+
+    # Initialize by loading dictionary and tokenize texts
+    with open(VOCAB_PATH, 'r') as f:
+        vocabulary = json.load(f)
+
+    st = SentenceTokenizer(vocabulary, 30)
+    tokens, _, _ = st.tokenize_sentences(test_sentences)
+
+    # Load model and run
+    model = torchmoji_emojis(weight_path=PRETRAINED_PATH)
+    prob = model(tokens)
+
+    # Find top emojis for each sentence
+    for i, t_prob in enumerate(list(prob)):
+        assert np.array_equal(top_elements(t_prob, 5), expected[i])
+
+
+def test_encode_texts():
+    """ Text encoding is stable.
+    """
+
+    TEST_SENTENCES = ['I love mom\'s cooking',
+                      'I love how you never reply back..',
+                      'I love cruising with my homies',
+                      'I love messing with yo mind!!',
+                      'I love you and now you\'re just gone..',
+                      'This is shit',
+                      'This is the shit']
+
+
+    maxlen = 30
+    batch_size = 32
+
+    with open(VOCAB_PATH, 'r') as f:
+        vocabulary = json.load(f)
+
+    st = SentenceTokenizer(vocabulary, maxlen)
+
+    print('Loading model from {}.'.format(PRETRAINED_PATH))
+    model = torchmoji_feature_encoding(PRETRAINED_PATH)
+    print(model)
+    tokenized, _, _ = st.tokenize_sentences(TEST_SENTENCES)
+    encoding = model(tokenized)
+
+    avg_across_sentences = np.around(np.mean(encoding, axis=0)[:5], 3)
+    assert np.allclose(avg_across_sentences, np.array([-0.023, 0.021, -0.037, -0.001, -0.005]))
+
+test_encode_texts()

tests/test_helper.py

@@ -0,0 +1,6 @@
+""" Module import helper.
+Modifies PATH in order to allow us to import the torchmoji directory.
+"""
+import sys
+from os.path import abspath, dirname
+sys.path.insert(0, dirname(dirname(abspath(__file__))))

tests/test_sentence_tokenizer.py

@@ -0,0 +1,113 @@
+from __future__ import absolute_import, print_function, division, unicode_literals
+import test_helper
+import json
+
+from torchmoji.sentence_tokenizer import SentenceTokenizer
+
+sentences = ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J']
+
+dicts = [
+    {'label': 0},
+    {'label': 1},
+    {'label': 2},
+    {'label': 3},
+    {'label': 4},
+    {'label': 5},
+    {'label': 6},
+    {'label': 7},
+    {'label': 8},
+    {'label': 9},
+    ]
+
+train_ind = [0, 5, 3, 6, 8]
+val_ind = [9, 2, 1]
+test_ind = [4, 7]
+
+with open('../model/vocabulary.json', 'r') as f:
+    vocab = json.load(f)
+
+def test_dataset_split_parameter():
+    """ Dataset is split in the desired ratios
+    """
+    split_parameter = [0.7, 0.1, 0.2]
+    st = SentenceTokenizer(vocab, 30)
+
+    result, result_dicts, _ = st.split_train_val_test(sentences, dicts,
+                                               split_parameter, extend_with=0)
+    train = result[0]
+    val = result[1]
+    test = result[2]
+
+    train_dicts = result_dicts[0]
+    val_dicts = result_dicts[1]
+    test_dicts = result_dicts[2]
+
+    assert len(train) == len(sentences) * split_parameter[0]
+    assert len(val) == len(sentences) * split_parameter[1]
+    assert len(test) == len(sentences) * split_parameter[2]
+
+    assert len(train_dicts) == len(dicts) * split_parameter[0]
+    assert len(val_dicts) == len(dicts) * split_parameter[1]
+    assert len(test_dicts) == len(dicts) * split_parameter[2]
+
+def test_dataset_split_explicit():
+    """ Dataset is split according to given indices
+    """
+    split_parameter = [train_ind, val_ind, test_ind]
+    st = SentenceTokenizer(vocab, 30)
+    tokenized, _, _ = st.tokenize_sentences(sentences)
+
+    result, result_dicts, added = st.split_train_val_test(sentences, dicts, split_parameter, extend_with=0)
+    train = result[0]
+    val = result[1]
+    test = result[2]
+
+    train_dicts = result_dicts[0]
+    val_dicts = result_dicts[1]
+    test_dicts = result_dicts[2]
+
+    tokenized = tokenized
+
+    for i, sentence in enumerate(sentences):
+        if i in train_ind:
+            assert tokenized[i] in train
+            assert dicts[i] in train_dicts
+        elif i in val_ind:
+            assert tokenized[i] in val
+            assert dicts[i] in val_dicts
+        elif i in test_ind:
+            assert tokenized[i] in test
+            assert dicts[i] in test_dicts
+
+    assert len(train) == len(train_ind)
+    assert len(val) == len(val_ind)
+    assert len(test) == len(test_ind)
+    assert len(train_dicts) == len(train_ind)
+    assert len(val_dicts) == len(val_ind)
+    assert len(test_dicts) == len(test_ind)
+
+def test_id_to_sentence():
+    """Tokenizing and converting back preserves the input.
+    """
+    vb = {'CUSTOM_MASK': 0,
+          'aasdf': 1000,
+          'basdf': 2000}
+
+    sentence = 'aasdf basdf basdf basdf'
+    st = SentenceTokenizer(vb, 30)
+    token, _, _ = st.tokenize_sentences([sentence])
+    assert st.to_sentence(token[0]) == sentence
+
+def test_id_to_sentence_with_unknown():
+    """Tokenizing and converting back preserves the input, except for unknowns.
+    """
+    vb = {'CUSTOM_MASK': 0,
+          'CUSTOM_UNKNOWN': 1,
+          'aasdf': 1000,
+          'basdf': 2000}
+
+    sentence = 'aasdf basdf ccc'
+    expected = 'aasdf basdf CUSTOM_UNKNOWN'
+    st = SentenceTokenizer(vb, 30)
+    token, _, _ = st.tokenize_sentences([sentence])
+    assert st.to_sentence(token[0]) == expected

tests/test_tokenizer.py

@@ -0,0 +1,167 @@
+# -*- coding: utf-8 -*-
+""" Tokenization tests.
+"""
+from __future__ import absolute_import, print_function, division, unicode_literals
+
+import sys
+from nose.tools import nottest
+from os.path import dirname, abspath
+sys.path.append(dirname(dirname(abspath(__file__))))
+from torchmoji.tokenizer import tokenize
+
+TESTS_NORMAL = [
+    ('200K words!', ['200', 'K', 'words', '!']),
+]
+
+TESTS_EMOJIS = [
+    ('i \U0001f496 you to the moon and back',
+     ['i', '\U0001f496', 'you', 'to', 'the', 'moon', 'and', 'back']),
+    ("i\U0001f496you to the \u2605's and back",
+     ['i', '\U0001f496', 'you', 'to', 'the',
+      '\u2605', "'", 's', 'and', 'back']),
+    ('~<3~', ['~', '<3', '~']),
+    ('<333', ['<333']),
+    (':-)', [':-)']),
+    ('>:-(', ['>:-(']),
+    ('\u266b\u266a\u2605\u2606\u2665\u2764\u2661',
+     ['\u266b', '\u266a', '\u2605', '\u2606',
+      '\u2665', '\u2764', '\u2661']),
+]
+
+TESTS_URLS = [
+    ('www.sample.com', ['www.sample.com']),
+    ('http://endless.horse', ['http://endless.horse']),
+    ('https://github.mit.ed', ['https://github.mit.ed']),
+]
+
+TESTS_TWITTER = [
+    ('#blacklivesmatter', ['#blacklivesmatter']),
+    ('#99_percent.', ['#99_percent', '.']),
+    ('the#99%', ['the', '#99', '%']),
+    ('@golden_zenith', ['@golden_zenith']),
+    ('@99_percent', ['@99_percent']),
+    ('[email protected]', ['[email protected]']),
+]
+
+TESTS_PHONE_NUMS = [
+    ('518)528-0252', ['518', ')', '528', '-', '0252']),
+    ('1200-0221-0234', ['1200', '-', '0221', '-', '0234']),
+    ('1200.0221.0234', ['1200', '.', '0221', '.', '0234']),
+]
+
+TESTS_DATETIME = [
+    ('15:00', ['15', ':', '00']),
+    ('2:00pm', ['2', ':', '00', 'pm']),
+    ('9/14/16', ['9', '/', '14', '/', '16']),
+]
+
+TESTS_CURRENCIES = [
+    ('517.933\xa3', ['517', '.', '933', '\xa3']),
+    ('$517.87', ['$', '517', '.', '87']),
+    ('1201.6598', ['1201', '.', '6598']),
+    ('120,6', ['120', ',', '6']),
+    ('10,00\u20ac', ['10', ',', '00', '\u20ac']),
+    ('1,000', ['1', ',', '000']),
+    ('1200pesos', ['1200', 'pesos']),
+]
+
+TESTS_NUM_SYM = [
+    ('5162f', ['5162', 'f']),
+    ('f5162', ['f', '5162']),
+    ('1203(', ['1203', '(']),
+    ('(1203)', ['(', '1203', ')']),
+    ('1200/', ['1200', '/']),
+    ('1200+', ['1200', '+']),
+    ('1202o-east', ['1202', 'o-east']),
+    ('1200r', ['1200', 'r']),
+    ('1200-1400', ['1200', '-', '1400']),
+    ('120/today', ['120', '/', 'today']),
+    ('today/120', ['today', '/', '120']),
+    ('120/5', ['120', '/', '5']),
+    ("120'/5", ['120', "'", '/', '5']),
+    ('120/5pro', ['120', '/', '5', 'pro']),
+    ("1200's,)", ['1200', "'", 's', ',', ')']),
+    ('120.76.218.207', ['120', '.', '76', '.', '218', '.', '207']),
+]
+
+TESTS_PUNCTUATION = [
+    ("don''t", ['don', "''", 't']),
+    ("don'tcha", ["don'tcha"]),
+    ('no?!?!;', ['no', '?', '!', '?', '!', ';']),
+    ('no??!!..', ['no', '??', '!!', '..']),
+    ('a.m.', ['a.m.']),
+    ('.s.u', ['.', 's', '.', 'u']),
+    ('!!i..n__', ['!!', 'i', '..', 'n', '__']),
+    ('lv(<3)w(3>)u Mr.!', ['lv', '(', '<3', ')', 'w', '(', '3',
+                            '>', ')', 'u', 'Mr.', '!']),
+    ('-->', ['--', '>']),
+    ('->', ['-', '>']),
+    ('<-', ['<', '-']),
+    ('<--', ['<', '--']),
+    ('hello (@person)', ['hello', '(', '@person', ')']),
+]
+
+
+def test_normal():
+    """ Normal/combined usage.
+    """
+    test_base(TESTS_NORMAL)
+
+
+def test_emojis():
+    """ Tokenizing emojis/emoticons/decorations.
+    """
+    test_base(TESTS_EMOJIS)
+
+
+def test_urls():
+    """ Tokenizing URLs.
+    """
+    test_base(TESTS_URLS)
+
+
+def test_twitter():
+    """ Tokenizing hashtags, mentions and emails.
+    """
+    test_base(TESTS_TWITTER)
+
+
+def test_phone_nums():
+    """ Tokenizing phone numbers.
+    """
+    test_base(TESTS_PHONE_NUMS)
+
+
+def test_datetime():
+    """ Tokenizing dates and times.
+    """
+    test_base(TESTS_DATETIME)
+
+
+def test_currencies():
+    """ Tokenizing currencies.
+    """
+    test_base(TESTS_CURRENCIES)
+
+
+def test_num_sym():
+    """ Tokenizing combinations of numbers and symbols.
+    """
+    test_base(TESTS_NUM_SYM)
+
+
+def test_punctuation():
+    """ Tokenizing punctuation and contractions.
+    """
+    test_base(TESTS_PUNCTUATION)
+
+
+@nottest
+def test_base(tests):
+    """ Base function for running tests.
+    """
+    for (test, expected) in tests:
+        actual = tokenize(test)
+        assert actual == expected, \
+            "Tokenization of \'{}\' failed, expected: {}, actual: {}"\
+            .format(test, expected, actual)

tests/test_word_generator.py

@@ -0,0 +1,73 @@
+# -*- coding: utf-8 -*-
+import sys
+from os.path import dirname, abspath
+sys.path.append(dirname(dirname(abspath(__file__))))
+from nose.tools import raises
+from torchmoji.word_generator import WordGenerator
+
+IS_PYTHON2 = int(sys.version[0]) == 2
+
+@raises(ValueError)
+def test_only_unicode_accepted():
+    """ Non-Unicode strings raise a ValueError.
+        In Python 3 all string are Unicode
+    """
+    if not IS_PYTHON2:
+        raise ValueError("You are using python 3 so this test should always pass")
+
+    sentences = [
+        u'Hello world',
+        u'I am unicode',
+        'I am not unicode',
+        ]
+
+    wg = WordGenerator(sentences)
+    for w in wg:
+        pass
+
+
+def test_unicode_sentences_ignored_if_set():
+    """ Strings with Unicode characters tokenize to empty array if they're not allowed.
+    """
+    sentence = [u'Dobrý den, jak se máš?']
+    wg = WordGenerator(sentence, allow_unicode_text=False)
+    assert wg.get_words(sentence[0]) == []
+
+
+def test_check_ascii():
+    """ check_ascii recognises ASCII words properly.
+        In Python 3 all string are Unicode
+    """
+    if not IS_PYTHON2:
+        return
+
+    wg = WordGenerator([])
+    assert wg.check_ascii('ASCII')
+    assert not wg.check_ascii('ščřžýá')
+    assert not wg.check_ascii('❤ ☀ ☆ ☂ ☻ ♞ ☯ ☭ ☢')
+
+
+def test_convert_unicode_word():
+    """ convert_unicode_word converts Unicode words correctly.
+    """
+    wg = WordGenerator([], allow_unicode_text=True)
+
+    result = wg.convert_unicode_word(u'č')
+    assert result == (True, u'\u010d'), '{}'.format(result)
+
+
+def test_convert_unicode_word_ignores_if_set():
+    """ convert_unicode_word ignores Unicode words if set.
+    """
+    wg = WordGenerator([], allow_unicode_text=False)
+
+    result = wg.convert_unicode_word(u'č')
+    assert result == (False, ''), '{}'.format(result)
+
+
+def test_convert_unicode_chars():
+    """ convert_unicode_word correctly converts accented characters.
+    """
+    wg = WordGenerator([], allow_unicode_text=True)
+    result = wg.convert_unicode_word(u'ěščřžýáíé')
+    assert result == (True, u'\u011b\u0161\u010d\u0159\u017e\xfd\xe1\xed\xe9'), '{}'.format(result)

torchmoji/.gitkeep

@@ -0,0 +1 @@
+ 

torchmoji/attlayer.py

@@ -0,0 +1,69 @@
+# -*- coding: utf-8 -*-
+""" Define the Attention Layer of the model.
+"""
+
+from __future__ import print_function, division
+
+import torch
+
+from torch.autograd import Variable
+from torch.nn import Module
+from torch.nn.parameter import Parameter
+
+class Attention(Module):
+    """
+    Computes a weighted average of the different channels across timesteps.
+    Uses 1 parameter pr. channel to compute the attention value for a single timestep.
+    """
+
+    def __init__(self, attention_size, return_attention=False):
+        """ Initialize the attention layer
+
+        # Arguments:
+            attention_size: Size of the attention vector.
+            return_attention: If true, output will include the weight for each input token
+                              used for the prediction
+
+        """
+        super(Attention, self).__init__()
+        self.return_attention = return_attention
+        self.attention_size = attention_size
+        self.attention_vector = Parameter(torch.FloatTensor(attention_size))
+
+    def __repr__(self):
+        s = '{name}({attention_size}, return attention={return_attention})'
+        return s.format(name=self.__class__.__name__, **self.__dict__)
+
+    def forward(self, inputs, input_lengths):
+        """ Forward pass.
+
+        # Arguments:
+            inputs (Torch.Variable): Tensor of input sequences
+            input_lengths (torch.LongTensor): Lengths of the sequences
+
+        # Return:
+            Tuple with (representations and attentions if self.return_attention else None).
+        """
+        logits = inputs.matmul(self.attention_vector)
+        unnorm_ai = (logits - logits.max()).exp()
+
+        # Compute a mask for the attention on the padded sequences
+        # See e.g. https://discuss.pytorch.org/t/self-attention-on-words-and-masking/5671/5
+        max_len = unnorm_ai.size(1)
+        idxes = torch.arange(0, max_len, out=torch.LongTensor(max_len)).unsqueeze(0)
+        if torch.cuda.is_available():
+            idxes = idxes.cuda()
+        mask = Variable((idxes < input_lengths.unsqueeze(1)).float())
+
+        # apply mask and renormalize attention scores (weights)
+        masked_weights = unnorm_ai * mask
+        att_sums = masked_weights.sum(dim=1, keepdim=True)  # sums per sequence
+        attentions = masked_weights.div(att_sums)
+
+        # apply attention weights
+        weighted = torch.mul(inputs, attentions.unsqueeze(-1).expand_as(inputs))
+
+        # get the final fixed vector representations of the sentences
+        representations = weighted.sum(dim=1)
+
+        return (representations, attentions if self.return_attention else None)

torchmoji/class_avg_finetuning.py

@@ -0,0 +1,315 @@
+# -*- coding: utf-8 -*-
+""" Class average finetuning functions. Before using any of these finetuning
+    functions, ensure that the model is set up with nb_classes=2.
+"""
+from __future__ import print_function
+
+import uuid
+from time import sleep
+import numpy as np
+
+import torch
+import torch.nn as nn
+import torch.optim as optim
+
+from torchmoji.global_variables import (
+    FINETUNING_METHODS,
+    WEIGHTS_DIR)
+from torchmoji.finetuning import (
+    freeze_layers,
+    get_data_loader,
+    fit_model,
+    train_by_chain_thaw,
+    find_f1_threshold)
+
+def relabel(y, current_label_nr, nb_classes):
+    """ Makes a binary classification for a specific class in a
+        multi-class dataset.
+
+    # Arguments:
+        y: Outputs to be relabelled.
+        current_label_nr: Current label number.
+        nb_classes: Total number of classes.
+
+    # Returns:
+        Relabelled outputs of a given multi-class dataset into a binary
+        classification dataset.
+    """
+
+    # Handling binary classification
+    if nb_classes == 2 and len(y.shape) == 1:
+        return y
+
+    y_new = np.zeros(len(y))
+    y_cut = y[:, current_label_nr]
+    label_pos = np.where(y_cut == 1)[0]
+    y_new[label_pos] = 1
+    return y_new
+
+
+def class_avg_finetune(model, texts, labels, nb_classes, batch_size,
+                       method, epoch_size=5000, nb_epochs=1000, embed_l2=1E-6,
+                       verbose=True):
+    """ Compiles and finetunes the given model.
+
+    # Arguments:
+        model: Model to be finetuned
+        texts: List of three lists, containing tokenized inputs for training,
+            validation and testing (in that order).
+        labels: List of three lists, containing labels for training,
+            validation and testing (in that order).
+        nb_classes: Number of classes in the dataset.
+        batch_size: Batch size.
+        method: Finetuning method to be used. For available methods, see
+            FINETUNING_METHODS in global_variables.py. Note that the model
+            should be defined accordingly (see docstring for torchmoji_transfer())
+        epoch_size: Number of samples in an epoch.
+        nb_epochs: Number of epochs. Doesn't matter much as early stopping is used.
+        embed_l2: L2 regularization for the embedding layer.
+        verbose: Verbosity flag.
+
+    # Returns:
+        Model after finetuning,
+        score after finetuning using the class average F1 metric.
+    """
+
+    if method not in FINETUNING_METHODS:
+        raise ValueError('ERROR (class_avg_tune_trainable): '
+                         'Invalid method parameter. '
+                         'Available options: {}'.format(FINETUNING_METHODS))
+
+    (X_train, y_train) = (texts[0], labels[0])
+    (X_val, y_val) = (texts[1], labels[1])
+    (X_test, y_test) = (texts[2], labels[2])
+
+    checkpoint_path = '{}/torchmoji-checkpoint-{}.bin' \
+                      .format(WEIGHTS_DIR, str(uuid.uuid4()))
+
+    f1_init_path = '{}/torchmoji-f1-init-{}.bin' \
+                   .format(WEIGHTS_DIR, str(uuid.uuid4()))
+
+    if method in ['last', 'new']:
+        lr = 0.001
+    elif method in ['full', 'chain-thaw']:
+        lr = 0.0001
+
+    loss_op = nn.BCEWithLogitsLoss()
+
+    # Freeze layers if using last
+    if method == 'last':
+        model = freeze_layers(model, unfrozen_keyword='output_layer')
+
+    # Define optimizer, for chain-thaw we define it later (after freezing)
+    if method == 'last':
+        adam = optim.Adam((p for p in model.parameters() if p.requires_grad), lr=lr)
+    elif method in ['full', 'new']:
+        # Add L2 regulation on embeddings only
+        special_params = [id(p) for p in model.embed.parameters()]
+        base_params = [p for p in model.parameters() if id(p) not in special_params and p.requires_grad]
+        embed_parameters = [p for p in model.parameters() if id(p) in special_params and p.requires_grad]
+        adam = optim.Adam([
+            {'params': base_params},
+            {'params': embed_parameters, 'weight_decay': embed_l2},
+            ], lr=lr)
+
+    # Training
+    if verbose:
+        print('Method:  {}'.format(method))
+        print('Classes: {}'.format(nb_classes))
+
+    if method == 'chain-thaw':
+        result = class_avg_chainthaw(model, nb_classes=nb_classes,
+                                     loss_op=loss_op,
+                                     train=(X_train, y_train),
+                                     val=(X_val, y_val),
+                                     test=(X_test, y_test),
+                                     batch_size=batch_size,
+                                     epoch_size=epoch_size,
+                                     nb_epochs=nb_epochs,
+                                     checkpoint_weight_path=checkpoint_path,
+                                     f1_init_weight_path=f1_init_path,
+                                     verbose=verbose)
+    else:
+        result = class_avg_tune_trainable(model, nb_classes=nb_classes,
+                                          loss_op=loss_op,
+                                          optim_op=adam,
+                                          train=(X_train, y_train),
+                                          val=(X_val, y_val),
+                                          test=(X_test, y_test),
+                                          epoch_size=epoch_size,
+                                          nb_epochs=nb_epochs,
+                                          batch_size=batch_size,
+                                          init_weight_path=f1_init_path,
+                                          checkpoint_weight_path=checkpoint_path,
+                                          verbose=verbose)
+    return model, result
+
+
+def prepare_labels(y_train, y_val, y_test, iter_i, nb_classes):
+    # Relabel into binary classification
+    y_train_new = relabel(y_train, iter_i, nb_classes)
+    y_val_new = relabel(y_val, iter_i, nb_classes)
+    y_test_new = relabel(y_test, iter_i, nb_classes)
+    return y_train_new, y_val_new, y_test_new
+
+def prepare_generators(X_train, y_train_new, X_val, y_val_new, batch_size, epoch_size):
+    # Create sample generators
+    # Make a fixed validation set to avoid fluctuations in validation
+    train_gen = get_data_loader(X_train, y_train_new, batch_size,
+                                extended_batch_sampler=True)
+    val_gen = get_data_loader(X_val, y_val_new, epoch_size,
+                              extended_batch_sampler=True)
+    X_val_resamp, y_val_resamp = next(iter(val_gen))
+    return train_gen, X_val_resamp, y_val_resamp
+
+
+def class_avg_tune_trainable(model, nb_classes, loss_op, optim_op, train, val, test,
+                             epoch_size, nb_epochs, batch_size,
+                             init_weight_path, checkpoint_weight_path, patience=5,
+                             verbose=True):
+    """ Finetunes the given model using the F1 measure.
+
+    # Arguments:
+        model: Model to be finetuned.
+        nb_classes: Number of classes in the given dataset.
+        train: Training data, given as a tuple of (inputs, outputs)
+        val: Validation data, given as a tuple of (inputs, outputs)
+        test: Testing data, given as a tuple of (inputs, outputs)
+        epoch_size: Number of samples in an epoch.
+        nb_epochs: Number of epochs.
+        batch_size: Batch size.
+        init_weight_path: Filepath where weights will be initially saved before
+            training each class. This file will be rewritten by the function.
+        checkpoint_weight_path: Filepath where weights will be checkpointed to
+            during training. This file will be rewritten by the function.
+        verbose: Verbosity flag.
+
+    # Returns:
+        F1 score of the trained model
+    """
+    total_f1 = 0
+    nb_iter = nb_classes if nb_classes > 2 else 1
+
+    # Unpack args
+    X_train, y_train = train
+    X_val, y_val = val
+    X_test, y_test = test
+
+    # Save and reload initial weights after running for
+    # each class to avoid learning across classes
+    torch.save(model.state_dict(), init_weight_path)
+    for i in range(nb_iter):
+        if verbose:
+            print('Iteration number {}/{}'.format(i+1, nb_iter))
+
+        model.load_state_dict(torch.load(init_weight_path))
+        y_train_new, y_val_new, y_test_new = prepare_labels(y_train, y_val,
+                                                            y_test, i, nb_classes)
+        train_gen, X_val_resamp, y_val_resamp = \
+            prepare_generators(X_train, y_train_new, X_val, y_val_new,
+                               batch_size, epoch_size)
+
+        if verbose:
+            print("Training..")
+        fit_model(model, loss_op, optim_op, train_gen, [(X_val_resamp, y_val_resamp)],
+                  nb_epochs, checkpoint_weight_path, patience, verbose=0)
+
+        # Reload the best weights found to avoid overfitting
+        # Wait a bit to allow proper closing of weights file
+        sleep(1)
+        model.load_state_dict(torch.load(checkpoint_weight_path))
+
+        # Evaluate
+        y_pred_val = model(X_val).cpu().numpy()
+        y_pred_test = model(X_test).cpu().numpy()
+
+        f1_test, best_t = find_f1_threshold(y_val_new, y_pred_val,
+                                            y_test_new, y_pred_test)
+        if verbose:
+            print('f1_test: {}'.format(f1_test))
+            print('best_t:  {}'.format(best_t))
+        total_f1 += f1_test
+
+    return total_f1 / nb_iter
+
+
+def class_avg_chainthaw(model, nb_classes, loss_op, train, val, test, batch_size,
+                        epoch_size, nb_epochs, checkpoint_weight_path,
+                        f1_init_weight_path, patience=5,
+                        initial_lr=0.001, next_lr=0.0001, verbose=True):
+    """ Finetunes given model using chain-thaw and evaluates using F1.
+        For a dataset with multiple classes, the model is trained once for
+        each class, relabeling those classes into a binary classification task.
+        The result is an average of all F1 scores for each class.
+
+    # Arguments:
+        model: Model to be finetuned.
+        nb_classes: Number of classes in the given dataset.
+        train: Training data, given as a tuple of (inputs, outputs)
+        val: Validation data, given as a tuple of (inputs, outputs)
+        test: Testing data, given as a tuple of (inputs, outputs)
+        batch_size: Batch size.
+        loss: Loss function to be used during training.
+        epoch_size: Number of samples in an epoch.
+        nb_epochs: Number of epochs.
+        checkpoint_weight_path: Filepath where weights will be checkpointed to
+            during training. This file will be rewritten by the function.
+        f1_init_weight_path: Filepath where weights will be saved to and
+            reloaded from before training each class. This ensures that
+            each class is trained independently. This file will be rewritten.
+        initial_lr: Initial learning rate. Will only be used for the first
+            training step (i.e. the softmax layer)
+        next_lr: Learning rate for every subsequent step.
+        seed: Random number generator seed.
+        verbose: Verbosity flag.
+
+    # Returns:
+        Averaged F1 score.
+    """
+
+    # Unpack args
+    X_train, y_train = train
+    X_val, y_val = val
+    X_test, y_test = test
+
+    total_f1 = 0
+    nb_iter = nb_classes if nb_classes > 2 else 1
+
+    torch.save(model.state_dict(), f1_init_weight_path)
+
+    for i in range(nb_iter):
+        if verbose:
+            print('Iteration number {}/{}'.format(i+1, nb_iter))
+
+        model.load_state_dict(torch.load(f1_init_weight_path))
+        y_train_new, y_val_new, y_test_new = prepare_labels(y_train, y_val,
+                                                            y_test, i, nb_classes)
+        train_gen, X_val_resamp, y_val_resamp = \
+                prepare_generators(X_train, y_train_new, X_val, y_val_new,
+                                   batch_size, epoch_size)
+
+        if verbose:
+            print("Training..")
+
+        # Train using chain-thaw
+        train_by_chain_thaw(model=model, train_gen=train_gen,
+                            val_gen=[(X_val_resamp, y_val_resamp)],
+                            loss_op=loss_op, patience=patience,
+                            nb_epochs=nb_epochs,
+                            checkpoint_path=checkpoint_weight_path,
+                            initial_lr=initial_lr, next_lr=next_lr,
+                            verbose=verbose)
+
+        # Evaluate
+        y_pred_val = model(X_val).cpu().numpy()
+        y_pred_test = model(X_test).cpu().numpy()
+
+        f1_test, best_t = find_f1_threshold(y_val_new, y_pred_val,
+                                            y_test_new, y_pred_test)
+
+        if verbose:
+            print('f1_test: {}'.format(f1_test))
+            print('best_t:  {}'.format(best_t))
+        total_f1 += f1_test
+
+    return total_f1 / nb_iter

torchmoji/create_vocab.py

@@ -0,0 +1,271 @@
+# -*- coding: utf-8 -*-
+from __future__ import print_function, division
+
+import glob
+import json
+import uuid
+from copy import deepcopy
+from collections import defaultdict, OrderedDict
+import numpy as np
+
+from torchmoji.filter_utils import is_special_token
+from torchmoji.word_generator import WordGenerator
+from torchmoji.global_variables import SPECIAL_TOKENS, VOCAB_PATH
+
+class VocabBuilder():
+    """ Create vocabulary with words extracted from sentences as fed from a
+        word generator.
+    """
+    def __init__(self, word_gen):
+        # initialize any new key with value of 0
+        self.word_counts = defaultdict(lambda: 0, {})
+        self.word_length_limit=30
+
+        for token in SPECIAL_TOKENS:
+            assert len(token) < self.word_length_limit
+            self.word_counts[token] = 0
+        self.word_gen = word_gen
+
+    def count_words_in_sentence(self, words):
+        """ Generates word counts for all tokens in the given sentence.
+
+        # Arguments:
+            words: Tokenized sentence whose words should be counted.
+        """
+        for word in words:
+            if 0 < len(word) and len(word) <= self.word_length_limit:
+                try:
+                    self.word_counts[word] += 1
+                except KeyError:
+                    self.word_counts[word] = 1
+
+    def save_vocab(self, path=None):
+        """ Saves the vocabulary into a file.
+
+        # Arguments:
+            path: Where the vocabulary should be saved. If not specified, a
+                  randomly generated filename is used instead.
+        """
+        dtype = ([('word','|S{}'.format(self.word_length_limit)),('count','int')])
+        np_dict = np.array(self.word_counts.items(), dtype=dtype)
+
+        # sort from highest to lowest frequency
+        np_dict[::-1].sort(order='count')
+        data = np_dict
+
+        if path is None:
+            path = str(uuid.uuid4())
+
+        np.savez_compressed(path, data=data)
+        print("Saved dict to {}".format(path))
+
+    def get_next_word(self):
+        """ Returns next tokenized sentence from the word geneerator.
+
+        # Returns:
+            List of strings, representing the next tokenized sentence.
+        """
+        return self.word_gen.__iter__().next()
+
+    def count_all_words(self):
+        """ Generates word counts for all words in all sentences of the word
+            generator.
+        """
+        for words, _ in self.word_gen:
+            self.count_words_in_sentence(words)
+
+class MasterVocab():
+    """ Combines vocabularies.
+    """
+    def __init__(self):
+
+        # initialize custom tokens
+        self.master_vocab = {}
+
+    def populate_master_vocab(self, vocab_path, min_words=1, force_appearance=None):
+        """ Populates the master vocabulary using all vocabularies found in the
+            given path. Vocabularies should be named *.npz. Expects the
+            vocabularies to be numpy arrays with counts. Normalizes the counts
+            and combines them.
+
+        # Arguments:
+            vocab_path: Path containing vocabularies to be combined.
+            min_words: Minimum amount of occurences a word must have in order
+                to be included in the master vocabulary.
+            force_appearance: Optional vocabulary filename that will be added
+                to the master vocabulary no matter what. This vocabulary must
+                be present in vocab_path.
+        """
+
+        paths = glob.glob(vocab_path + '*.npz')
+        sizes = {path: 0 for path in paths}
+        dicts = {path: {} for path in paths}
+
+        # set up and get sizes of individual dictionaries
+        for path in paths:
+            np_data = np.load(path)['data']
+
+            for entry in np_data:
+                word, count = entry
+                if count < min_words:
+                    continue
+                if is_special_token(word):
+                    continue
+                dicts[path][word] = count
+
+            sizes[path] = sum(dicts[path].values())
+            print('Overall word count for {} -> {}'.format(path, sizes[path]))
+            print('Overall word number for {} -> {}'.format(path, len(dicts[path])))
+
+        vocab_of_max_size = max(sizes, key=sizes.get)
+        max_size = sizes[vocab_of_max_size]
+        print('Min: {}, {}, {}'.format(sizes, vocab_of_max_size, max_size))
+
+        # can force one vocabulary to always be present
+        if force_appearance is not None:
+            force_appearance_path = [p for p in paths if force_appearance in p][0]
+            force_appearance_vocab = deepcopy(dicts[force_appearance_path])
+            print(force_appearance_path)
+        else:
+            force_appearance_path, force_appearance_vocab = None, None
+
+        # normalize word counts before inserting into master dict
+        for path in paths:
+            normalization_factor = max_size / sizes[path]
+            print('Norm factor for path {} -> {}'.format(path, normalization_factor))
+
+            for word in dicts[path]:
+                if is_special_token(word):
+                    print("SPECIAL - ", word)
+                    continue
+                normalized_count = dicts[path][word] * normalization_factor
+
+                # can force one vocabulary to always be present
+                if force_appearance_vocab is not None:
+                    try:
+                        force_word_count = force_appearance_vocab[word]
+                    except KeyError:
+                        continue
+                    #if force_word_count < 5:
+                        #continue
+
+                if word in self.master_vocab:
+                    self.master_vocab[word] += normalized_count
+                else:
+                    self.master_vocab[word] = normalized_count
+
+        print('Size of master_dict {}'.format(len(self.master_vocab)))
+        print("Hashes for master dict: {}".format(
+            len([w for w in self.master_vocab if '#' in w[0]])))
+
+    def save_vocab(self, path_count, path_vocab, word_limit=100000):
+        """ Saves the master vocabulary into a file.
+        """
+
+        # reserve space for 10 special tokens
+        words = OrderedDict()
+        for token in SPECIAL_TOKENS:
+            # store -1 instead of np.inf, which can overflow
+            words[token] = -1
+
+        # sort words by frequency
+        desc_order = OrderedDict(sorted(self.master_vocab.items(),
+                                 key=lambda kv: kv[1], reverse=True))
+        words.update(desc_order)
+
+        # use encoding of up to 30 characters (no token conversions)
+        # use float to store large numbers (we don't care about precision loss)
+        np_vocab = np.array(words.items(),
+                            dtype=([('word','|S30'),('count','float')]))
+
+        # output count for debugging
+        counts = np_vocab[:word_limit]
+        np.savez_compressed(path_count, counts=counts)
+
+        # output the index of each word for easy lookup
+        final_words = OrderedDict()
+        for i, w in enumerate(words.keys()[:word_limit]):
+            final_words.update({w:i})
+        with open(path_vocab, 'w') as f:
+            f.write(json.dumps(final_words, indent=4, separators=(',', ': ')))
+
+
+def all_words_in_sentences(sentences):
+    """ Extracts all unique words from a given list of sentences.
+
+    # Arguments:
+        sentences: List or word generator of sentences to be processed.
+
+    # Returns:
+        List of all unique words contained in the given sentences.
+    """
+    vocab = []
+    if isinstance(sentences, WordGenerator):
+        sentences = [s for s, _ in sentences]
+
+    for sentence in sentences:
+        for word in sentence:
+            if word not in vocab:
+                vocab.append(word)
+
+    return vocab
+
+
+def extend_vocab_in_file(vocab, max_tokens=10000, vocab_path=VOCAB_PATH):
+    """ Extends JSON-formatted vocabulary with words from vocab that are not
+        present in the current vocabulary. Adds up to max_tokens words.
+        Overwrites file in vocab_path.
+
+    # Arguments:
+        new_vocab: Vocabulary to be added. MUST have word_counts populated, i.e.
+            must have run count_all_words() previously.
+        max_tokens: Maximum number of words to be added.
+        vocab_path: Path to the vocabulary json which is to be extended.
+    """
+    try:
+        with open(vocab_path, 'r') as f:
+            current_vocab = json.load(f)
+    except IOError:
+        print('Vocabulary file not found, expected at ' + vocab_path)
+        return
+
+    extend_vocab(current_vocab, vocab, max_tokens)
+
+    # Save back to file
+    with open(vocab_path, 'w') as f:
+        json.dump(current_vocab, f, sort_keys=True, indent=4, separators=(',',': '))
+
+
+def extend_vocab(current_vocab, new_vocab, max_tokens=10000):
+    """ Extends current vocabulary with words from vocab that are not
+        present in the current vocabulary. Adds up to max_tokens words.
+
+    # Arguments:
+        current_vocab: Current dictionary of tokens.
+        new_vocab: Vocabulary to be added. MUST have word_counts populated, i.e.
+            must have run count_all_words() previously.
+        max_tokens: Maximum number of words to be added.
+
+    # Returns:
+        How many new tokens have been added.
+    """
+    if max_tokens < 0:
+        max_tokens = 10000
+
+    words = OrderedDict()
+
+    # sort words by frequency
+    desc_order = OrderedDict(sorted(new_vocab.word_counts.items(),
+                                key=lambda kv: kv[1], reverse=True))
+    words.update(desc_order)
+
+    base_index = len(current_vocab.keys())
+    added = 0
+    for word in words:
+        if added >= max_tokens:
+            break
+        if word not in current_vocab.keys():
+            current_vocab[word] = base_index + added
+            added += 1
+
+    return added

torchmoji/filter_input.py

@@ -0,0 +1,36 @@
+# -*- coding: utf-8 -*-
+from __future__ import print_function, division
+import codecs
+import csv
+import numpy as np
+from emoji import UNICODE_EMOJI
+
+def read_english(path="english_words.txt", add_emojis=True):
+    # read english words for filtering (includes emojis as part of set)
+    english = set()
+    with codecs.open(path, "r", "utf-8") as f:
+        for line in f:
+            line = line.strip().lower().replace('\n', '')
+            if len(line):
+                english.add(line)
+    if add_emojis:
+        for e in UNICODE_EMOJI:
+            english.add(e)
+    return english
+
+def read_wanted_emojis(path="wanted_emojis.csv"):
+    emojis = []
+    with open(path, 'rb') as f:
+        reader = csv.reader(f)
+        for line in reader:
+            line = line[0].strip().replace('\n', '')
+            line = line.decode('unicode-escape')
+            emojis.append(line)
+    return emojis
+
+def read_non_english_users(path="unwanted_users.npz"):
+    try:
+        neu_set = set(np.load(path)['userids'])
+    except IOError:
+        neu_set = set()
+    return neu_set

torchmoji/filter_utils.py

@@ -0,0 +1,191 @@
+
+# -*- coding: utf-8 -*-
+from __future__ import print_function, division, unicode_literals
+import sys
+import re
+import string
+import emoji
+from itertools import groupby
+
+import numpy as np
+from torchmoji.tokenizer import RE_MENTION, RE_URL
+from torchmoji.global_variables import SPECIAL_TOKENS
+
+IS_PYTHON2 = int(sys.version[0]) == 2
+chr_ = unichr if IS_PYTHON2 else chr
+
+AtMentionRegex = re.compile(RE_MENTION)
+urlRegex = re.compile(RE_URL)
+
+# from http://bit.ly/2rdjgjE (UTF-8 encodings and Unicode chars)
+VARIATION_SELECTORS = [ '\ufe00',
+                        '\ufe01',
+                        '\ufe02',
+                        '\ufe03',
+                        '\ufe04',
+                        '\ufe05',
+                        '\ufe06',
+                        '\ufe07',
+                        '\ufe08',
+                        '\ufe09',
+                        '\ufe0a',
+                        '\ufe0b',
+                        '\ufe0c',
+                        '\ufe0d',
+                        '\ufe0e',
+                        '\ufe0f']
+
+# from https://stackoverflow.com/questions/92438/stripping-non-printable-characters-from-a-string-in-python
+ALL_CHARS = (chr_(i) for i in range(sys.maxunicode))
+CONTROL_CHARS = ''.join(map(chr_, list(range(0,32)) + list(range(127,160))))
+CONTROL_CHAR_REGEX = re.compile('[%s]' % re.escape(CONTROL_CHARS))
+
+def is_special_token(word):
+    equal = False
+    for spec in SPECIAL_TOKENS:
+        if word == spec:
+            equal = True
+            break
+    return equal
+
+def mostly_english(words, english, pct_eng_short=0.5, pct_eng_long=0.6, ignore_special_tokens=True, min_length=2):
+    """ Ensure text meets threshold for containing English words """
+
+    n_words = 0
+    n_english = 0
+
+    if english is None:
+        return True, 0, 0
+
+    for w in words:
+        if len(w) < min_length:
+            continue
+        if punct_word(w):
+            continue
+        if ignore_special_tokens and is_special_token(w):
+            continue
+        n_words += 1
+        if w in english:
+            n_english += 1
+
+    if n_words < 2:
+        return True, n_words, n_english
+    if n_words < 5:
+        valid_english = n_english >= n_words * pct_eng_short
+    else:
+        valid_english = n_english >= n_words * pct_eng_long
+    return valid_english, n_words, n_english
+
+def correct_length(words, min_words, max_words, ignore_special_tokens=True):
+    """ Ensure text meets threshold for containing English words
+        and that it's within the min and max words limits. """
+
+    if min_words is None:
+        min_words = 0
+
+    if max_words is None:
+        max_words = 99999
+
+    n_words = 0
+    for w in words:
+        if punct_word(w):
+            continue
+        if ignore_special_tokens and is_special_token(w):
+            continue
+        n_words += 1
+    valid = min_words <= n_words and n_words <= max_words
+    return valid
+
+def punct_word(word, punctuation=string.punctuation):
+    return all([True if c in punctuation else False for c in word])
+
+def load_non_english_user_set():
+    non_english_user_set = set(np.load('uids.npz')['data'])
+    return non_english_user_set
+
+def non_english_user(userid, non_english_user_set):
+    neu_found = int(userid) in non_english_user_set
+    return neu_found
+
+def separate_emojis_and_text(text):
+    emoji_chars = []
+    non_emoji_chars = []
+    for c in text:
+        if c in emoji.UNICODE_EMOJI:
+            emoji_chars.append(c)
+        else:
+            non_emoji_chars.append(c)
+    return ''.join(emoji_chars), ''.join(non_emoji_chars)
+
+def extract_emojis(text, wanted_emojis):
+    text = remove_variation_selectors(text)
+    return [c for c in text if c in wanted_emojis]
+
+def remove_variation_selectors(text):
+    """ Remove styling glyph variants for Unicode characters.
+        For instance, remove skin color from emojis.
+    """
+    for var in VARIATION_SELECTORS:
+        text = text.replace(var, '')
+    return text
+
+def shorten_word(word):
+    """ Shorten groupings of 3+ identical consecutive chars to 2, e.g. '!!!!' --> '!!'
+    """
+
+    # only shorten ASCII words
+    try:
+        word.decode('ascii')
+    except (UnicodeDecodeError, UnicodeEncodeError, AttributeError) as e:
+        return word
+
+    # must have at least 3 char to be shortened
+    if len(word) < 3:
+        return word
+
+    # find groups of 3+ consecutive letters
+    letter_groups = [list(g) for k, g in groupby(word)]
+    triple_or_more = [''.join(g) for g in letter_groups if len(g) >= 3]
+    if len(triple_or_more) == 0:
+        return word
+
+    # replace letters to find the short word
+    short_word = word
+    for trip in triple_or_more:
+        short_word = short_word.replace(trip, trip[0]*2)
+
+    return short_word
+
+def detect_special_tokens(word):
+    try:
+        int(word)
+        word = SPECIAL_TOKENS[4]
+    except ValueError:
+        if AtMentionRegex.findall(word):
+            word = SPECIAL_TOKENS[2]
+        elif urlRegex.findall(word):
+            word = SPECIAL_TOKENS[3]
+    return word
+
+def process_word(word):
+    """ Shortening and converting the word to a special token if relevant.
+    """
+    word = shorten_word(word)
+    word = detect_special_tokens(word)
+    return word
+
+def remove_control_chars(text):
+    return CONTROL_CHAR_REGEX.sub('', text)
+
+def convert_nonbreaking_space(text):
+    # ugly hack handling non-breaking space no matter how badly it's been encoded in the input
+    for r in ['\\\\xc2', '\\xc2', '\xc2', '\\\\xa0', '\\xa0', '\xa0']:
+        text = text.replace(r, ' ')
+    return text
+
+def convert_linebreaks(text):
+    # ugly hack handling non-breaking space no matter how badly it's been encoded in the input
+    # space around to ensure proper tokenization
+    for r in ['\\\\n', '\\n', '\n', '\\\\r', '\\r', '\r', '<br>']:
+        text = text.replace(r, ' ' + SPECIAL_TOKENS[5] + ' ')
+    return text

torchmoji/finetuning.py

@@ -0,0 +1,661 @@
+# -*- coding: utf-8 -*-
+""" Finetuning functions for doing transfer learning to new datasets.
+"""
+from __future__ import print_function
+
+import sys
+import uuid
+from time import sleep
+from io import open
+
+import math
+import pickle
+import numpy as np
+
+import torch
+import torch.nn as nn
+import torch.optim as optim
+from torch.autograd import Variable
+from torch.utils.data import Dataset, DataLoader
+from torch.utils.data.sampler import BatchSampler, SequentialSampler
+from torch.nn.utils import clip_grad_norm
+
+from sklearn.metrics import f1_score
+
+from torchmoji.global_variables import (FINETUNING_METHODS,
+                                               FINETUNING_METRICS,
+                                               WEIGHTS_DIR)
+from torchmoji.tokenizer import tokenize
+from torchmoji.sentence_tokenizer import SentenceTokenizer
+
+IS_PYTHON2 = int(sys.version[0]) == 2
+unicode_ = unicode if IS_PYTHON2 else str
+
+def load_benchmark(path, vocab, extend_with=0):
+    """ Loads the given benchmark dataset.
+
+        Tokenizes the texts using the provided vocabulary, extending it with
+        words from the training dataset if extend_with > 0. Splits them into
+        three lists: training, validation and testing (in that order).
+
+        Also calculates the maximum length of the texts and the
+        suggested batch_size.
+
+    # Arguments:
+        path: Path to the dataset to be loaded.
+        vocab: Vocabulary to be used for tokenizing texts.
+        extend_with: If > 0, the vocabulary will be extended with up to
+            extend_with tokens from the training set before tokenizing.
+
+    # Returns:
+        A dictionary with the following fields:
+            texts: List of three lists, containing tokenized inputs for
+                training, validation and testing (in that order).
+            labels: List of three lists, containing labels for training,
+                validation and testing (in that order).
+            added: Number of tokens added to the vocabulary.
+            batch_size: Batch size.
+            maxlen: Maximum length of an input.
+    """
+    # Pre-processing dataset
+    with open(path, 'rb') as dataset:
+        if IS_PYTHON2:
+            data = pickle.load(dataset)
+        else:
+            data = pickle.load(dataset, fix_imports=True)
+
+    # Decode data
+    try:
+        texts = [unicode_(x) for x in data['texts']]
+    except UnicodeDecodeError:
+        texts = [x.decode('utf-8') for x in data['texts']]
+
+    # Extract labels
+    labels = [x['label'] for x in data['info']]
+
+    batch_size, maxlen = calculate_batchsize_maxlen(texts)
+
+    st = SentenceTokenizer(vocab, maxlen)
+
+    # Split up dataset. Extend the existing vocabulary with up to extend_with
+    # tokens from the training dataset.
+    texts, labels, added = st.split_train_val_test(texts,
+                                                   labels,
+                                                   [data['train_ind'],
+                                                    data['val_ind'],
+                                                    data['test_ind']],
+                                                   extend_with=extend_with)
+    return {'texts': texts,
+            'labels': labels,
+            'added': added,
+            'batch_size': batch_size,
+            'maxlen': maxlen}
+
+
+def calculate_batchsize_maxlen(texts):
+    """ Calculates the maximum length in the provided texts and a suitable
+        batch size. Rounds up maxlen to the nearest multiple of ten.
+
+    # Arguments:
+        texts: List of inputs.
+
+    # Returns:
+        Batch size,
+        max length
+    """
+    def roundup(x):
+        return int(math.ceil(x / 10.0)) * 10
+
+    # Calculate max length of sequences considered
+    # Adjust batch_size accordingly to prevent GPU overflow
+    lengths = [len(tokenize(t)) for t in texts]
+    maxlen = roundup(np.percentile(lengths, 80.0))
+    batch_size = 250 if maxlen <= 100 else 50
+    return batch_size, maxlen
+
+
+
+def freeze_layers(model, unfrozen_types=[], unfrozen_keyword=None):
+    """ Freezes all layers in the given model, except for ones that are
+        explicitly specified to not be frozen.
+
+    # Arguments:
+        model: Model whose layers should be modified.
+        unfrozen_types: List of layer types which shouldn't be frozen.
+        unfrozen_keyword: Name keywords of layers that shouldn't be frozen.
+
+    # Returns:
+        Model with the selected layers frozen.
+    """
+    # Get trainable modules
+    trainable_modules = [(n, m) for n, m in model.named_children() if len([id(p) for p in m.parameters()]) !=  0]
+    for name, module in trainable_modules:
+        trainable = (any(typ in str(module) for typ in unfrozen_types) or
+                     (unfrozen_keyword is not None and unfrozen_keyword.lower() in name.lower()))
+        change_trainable(module, trainable, verbose=False)
+    return model
+
+
+def change_trainable(module, trainable, verbose=False):
+    """ Helper method that freezes or unfreezes a given layer.
+
+    # Arguments:
+        module: Module to be modified.
+        trainable: Whether the layer should be frozen or unfrozen.
+        verbose: Verbosity flag.
+    """
+    
+    if verbose: print('Changing MODULE', module, 'to trainable =', trainable)
+    for name, param in module.named_parameters():
+        if verbose: print('Setting weight', name, 'to trainable =', trainable)
+        param.requires_grad = trainable
+
+    if verbose:
+        action = 'Unfroze' if trainable else 'Froze'
+        if verbose: print("{} {}".format(action, module))
+
+
+def find_f1_threshold(model, val_gen, test_gen, average='binary'):
+    """ Choose a threshold for F1 based on the validation dataset
+        (see https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4442797/
+        for details on why to find another threshold than simply 0.5)
+
+    # Arguments:
+        model: pyTorch model
+        val_gen: Validation set dataloader.
+        test_gen: Testing set dataloader.
+
+    # Returns:
+        F1 score for the given data and
+        the corresponding F1 threshold
+    """
+    thresholds = np.arange(0.01, 0.5, step=0.01)
+    f1_scores = []
+
+    model.eval()
+    val_out = [(y, model(X)) for X, y in val_gen]
+    y_val, y_pred_val = (list(t) for t in zip(*val_out))
+
+    test_out = [(y, model(X)) for X, y in test_gen]
+    y_test, y_pred_test = (list(t) for t in zip(*val_out))
+
+    for t in thresholds:
+        y_pred_val_ind = (y_pred_val > t)
+        f1_val = f1_score(y_val, y_pred_val_ind, average=average)
+        f1_scores.append(f1_val)
+
+    best_t = thresholds[np.argmax(f1_scores)]
+    y_pred_ind = (y_pred_test > best_t)
+    f1_test = f1_score(y_test, y_pred_ind, average=average)
+    return f1_test, best_t
+
+
+def finetune(model, texts, labels, nb_classes, batch_size, method,
+             metric='acc', epoch_size=5000, nb_epochs=1000, embed_l2=1E-6,
+             verbose=1):
+    """ Compiles and finetunes the given pytorch model.
+
+    # Arguments:
+        model: Model to be finetuned
+        texts: List of three lists, containing tokenized inputs for training,
+            validation and testing (in that order).
+        labels: List of three lists, containing labels for training,
+            validation and testing (in that order).
+        nb_classes: Number of classes in the dataset.
+        batch_size: Batch size.
+        method: Finetuning method to be used. For available methods, see
+            FINETUNING_METHODS in global_variables.py.
+        metric: Evaluation metric to be used. For available metrics, see
+            FINETUNING_METRICS in global_variables.py.
+        epoch_size: Number of samples in an epoch.
+        nb_epochs: Number of epochs. Doesn't matter much as early stopping is used.
+        embed_l2: L2 regularization for the embedding layer.
+        verbose: Verbosity flag.
+
+    # Returns:
+        Model after finetuning,
+        score after finetuning using the provided metric.
+    """
+
+    if method not in FINETUNING_METHODS:
+        raise ValueError('ERROR (finetune): Invalid method parameter. '
+                         'Available options: {}'.format(FINETUNING_METHODS))
+    if metric not in FINETUNING_METRICS:
+        raise ValueError('ERROR (finetune): Invalid metric parameter. '
+                         'Available options: {}'.format(FINETUNING_METRICS))
+
+    train_gen = get_data_loader(texts[0], labels[0], batch_size,
+                                extended_batch_sampler=True, epoch_size=epoch_size)
+    val_gen = get_data_loader(texts[1], labels[1], batch_size,
+                              extended_batch_sampler=False)
+    test_gen = get_data_loader(texts[2], labels[2], batch_size,
+                              extended_batch_sampler=False)
+
+    checkpoint_path = '{}/torchmoji-checkpoint-{}.bin' \
+                      .format(WEIGHTS_DIR, str(uuid.uuid4()))
+
+    if method in ['last', 'new']:
+        lr = 0.001
+    elif method in ['full', 'chain-thaw']:
+        lr = 0.0001
+
+    loss_op = nn.BCEWithLogitsLoss() if nb_classes <= 2 \
+         else nn.CrossEntropyLoss()
+
+    # Freeze layers if using last
+    if method == 'last':
+        model = freeze_layers(model, unfrozen_keyword='output_layer')
+
+    # Define optimizer, for chain-thaw we define it later (after freezing)
+    if method == 'last':
+        adam = optim.Adam((p for p in model.parameters() if p.requires_grad), lr=lr)
+    elif method in ['full', 'new']:
+        # Add L2 regulation on embeddings only
+        embed_params_id = [id(p) for p in model.embed.parameters()]
+        output_layer_params_id = [id(p) for p in model.output_layer.parameters()]
+        base_params = [p for p in model.parameters()
+                       if id(p) not in embed_params_id and id(p) not in output_layer_params_id and p.requires_grad]
+        embed_params = [p for p in model.parameters() if id(p) in embed_params_id and p.requires_grad]
+        output_layer_params = [p for p in model.parameters() if id(p) in output_layer_params_id and p.requires_grad]
+        adam = optim.Adam([
+            {'params': base_params},
+            {'params': embed_params, 'weight_decay': embed_l2},
+            {'params': output_layer_params, 'lr': 0.001},
+            ], lr=lr)
+
+    # Training
+    if verbose:
+        print('Method:  {}'.format(method))
+        print('Metric:  {}'.format(metric))
+        print('Classes: {}'.format(nb_classes))
+
+    if method == 'chain-thaw':
+        result = chain_thaw(model, train_gen, val_gen, test_gen, nb_epochs, checkpoint_path, loss_op, embed_l2=embed_l2,
+                            evaluate=metric, verbose=verbose)
+    else:
+        result = tune_trainable(model, loss_op, adam, train_gen, val_gen, test_gen, nb_epochs, checkpoint_path,
+                                evaluate=metric, verbose=verbose)
+    return model, result
+
+
+def tune_trainable(model, loss_op, optim_op, train_gen, val_gen, test_gen,
+                   nb_epochs, checkpoint_path, patience=5, evaluate='acc',
+                   verbose=2):
+    """ Finetunes the given model using the accuracy measure.
+
+    # Arguments:
+        model: Model to be finetuned.
+        nb_classes: Number of classes in the given dataset.
+        train: Training data, given as a tuple of (inputs, outputs)
+        val: Validation data, given as a tuple of (inputs, outputs)
+        test: Testing data, given as a tuple of (inputs, outputs)
+        epoch_size: Number of samples in an epoch.
+        nb_epochs: Number of epochs.
+        batch_size: Batch size.
+        checkpoint_weight_path: Filepath where weights will be checkpointed to
+            during training. This file will be rewritten by the function.
+        patience: Patience for callback methods.
+        evaluate: Evaluation method to use. Can be 'acc' or 'weighted_f1'.
+        verbose: Verbosity flag.
+
+    # Returns:
+        Accuracy of the trained model, ONLY if 'evaluate' is set.
+    """
+    if verbose:
+        print("Trainable weights: {}".format([n for n, p in model.named_parameters() if p.requires_grad]))
+        print("Training...")
+        if evaluate == 'acc':
+            print("Evaluation on test set prior training:", evaluate_using_acc(model, test_gen))
+        elif evaluate == 'weighted_f1':
+            print("Evaluation on test set prior training:", evaluate_using_weighted_f1(model, test_gen, val_gen))
+
+    fit_model(model, loss_op, optim_op, train_gen, val_gen, nb_epochs, checkpoint_path, patience)
+
+    # Reload the best weights found to avoid overfitting
+    # Wait a bit to allow proper closing of weights file
+    sleep(1)
+    model.load_state_dict(torch.load(checkpoint_path))
+    if verbose >= 2:
+        print("Loaded weights from {}".format(checkpoint_path))
+
+    if evaluate == 'acc':
+        return evaluate_using_acc(model, test_gen)
+    elif evaluate == 'weighted_f1':
+        return evaluate_using_weighted_f1(model, test_gen, val_gen)
+
+
+def evaluate_using_weighted_f1(model, test_gen, val_gen):
+    """ Evaluation function using macro weighted F1 score.
+
+    # Arguments:
+        model: Model to be evaluated.
+        X_test: Inputs of the testing set.
+        y_test: Outputs of the testing set.
+        X_val: Inputs of the validation set.
+        y_val: Outputs of the validation set.
+        batch_size: Batch size.
+
+    # Returns:
+        Weighted F1 score of the given model.
+    """
+    # Evaluate on test and val data
+    f1_test, _ = find_f1_threshold(model, test_gen, val_gen, average='weighted_f1')
+    return f1_test
+
+
+def evaluate_using_acc(model, test_gen):
+    """ Evaluation function using accuracy.
+
+    # Arguments:
+        model: Model to be evaluated.
+        test_gen: Testing data iterator (DataLoader)
+
+    # Returns:
+        Accuracy of the given model.
+    """
+
+    # Validate on test_data
+    model.eval()
+    correct_count = 0.0
+    total_y = sum(len(y) for _, y in test_gen)
+    for i, data in enumerate(test_gen):
+        x, y = data
+        outs = model(x)
+        pred = (outs >= 0).long()
+        added_counts = (pred == y).double().sum()
+        correct_count += added_counts
+    return correct_count/total_y
+
+
+def chain_thaw(model, train_gen, val_gen, test_gen, nb_epochs, checkpoint_path, loss_op,
+               patience=5, initial_lr=0.001, next_lr=0.0001, embed_l2=1E-6, evaluate='acc', verbose=1):
+    """ Finetunes given model using chain-thaw and evaluates using accuracy.
+
+    # Arguments:
+        model: Model to be finetuned.
+        train: Training data, given as a tuple of (inputs, outputs)
+        val: Validation data, given as a tuple of (inputs, outputs)
+        test: Testing data, given as a tuple of (inputs, outputs)
+        batch_size: Batch size.
+        loss: Loss function to be used during training.
+        epoch_size: Number of samples in an epoch.
+        nb_epochs: Number of epochs.
+        checkpoint_weight_path: Filepath where weights will be checkpointed to
+            during training. This file will be rewritten by the function.
+        initial_lr: Initial learning rate. Will only be used for the first
+            training step (i.e. the output_layer layer)
+        next_lr: Learning rate for every subsequent step.
+        seed: Random number generator seed.
+        verbose: Verbosity flag.
+        evaluate: Evaluation method to use. Can be 'acc' or 'weighted_f1'.
+
+    # Returns:
+        Accuracy of the finetuned model.
+    """
+    if verbose:
+        print('Training..')
+
+    # Train using chain-thaw
+    train_by_chain_thaw(model, train_gen, val_gen, loss_op, patience, nb_epochs, checkpoint_path,
+                        initial_lr, next_lr, embed_l2, verbose)
+
+    if evaluate == 'acc':
+        return evaluate_using_acc(model, test_gen)
+    elif evaluate == 'weighted_f1':
+        return evaluate_using_weighted_f1(model, test_gen, val_gen)
+
+
+def train_by_chain_thaw(model, train_gen, val_gen, loss_op, patience, nb_epochs, checkpoint_path,
+                        initial_lr=0.001, next_lr=0.0001, embed_l2=1E-6, verbose=1):
+    """ Finetunes model using the chain-thaw method.
+
+    This is done as follows:
+    1) Freeze every layer except the last (output_layer) layer and train it.
+    2) Freeze every layer except the first layer and train it.
+    3) Freeze every layer except the second etc., until the second last layer.
+    4) Unfreeze all layers and train entire model.
+
+    # Arguments:
+        model: Model to be trained.
+        train_gen: Training sample generator.
+        val_data: Validation data.
+        loss: Loss function to be used.
+        finetuning_args: Training early stopping and checkpoint saving parameters
+        epoch_size: Number of samples in an epoch.
+        nb_epochs: Number of epochs.
+        checkpoint_weight_path: Where weight checkpoints should be saved.
+        batch_size: Batch size.
+        initial_lr: Initial learning rate. Will only be used for the first
+            training step (i.e. the output_layer layer)
+        next_lr: Learning rate for every subsequent step.
+        verbose: Verbosity flag.
+    """
+    # Get trainable layers
+    layers = [m for m in model.children() if len([id(p) for p in m.parameters()]) !=  0]
+
+    # Bring last layer to front
+    layers.insert(0, layers.pop(len(layers) - 1))
+
+    # Add None to the end to signify finetuning all layers
+    layers.append(None)
+
+    lr = None
+    # Finetune each layer one by one and finetune all of them at once
+    # at the end
+    for layer in layers:
+        if lr is None:
+            lr = initial_lr
+        elif lr == initial_lr:
+            lr = next_lr
+
+        # Freeze all except current layer
+        for _layer in layers:
+            if _layer is not None:
+                trainable = _layer == layer or layer is None
+                change_trainable(_layer, trainable=trainable, verbose=False)
+
+        # Verify we froze the right layers
+        for _layer in model.children():
+            assert all(p.requires_grad == (_layer == layer) for p in _layer.parameters()) or layer is None
+
+        if verbose:
+            if layer is None:
+                print('Finetuning all layers')
+            else:
+                print('Finetuning {}'.format(layer))
+
+        special_params = [id(p) for p in model.embed.parameters()]
+        base_params = [p for p in model.parameters() if id(p) not in special_params and p.requires_grad]
+        embed_parameters = [p for p in model.parameters() if id(p) in special_params and p.requires_grad]
+        adam = optim.Adam([
+            {'params': base_params},
+            {'params': embed_parameters, 'weight_decay': embed_l2},
+            ], lr=lr)
+
+        fit_model(model, loss_op, adam, train_gen, val_gen, nb_epochs,
+                  checkpoint_path, patience)
+
+        # Reload the best weights found to avoid overfitting
+        # Wait a bit to allow proper closing of weights file
+        sleep(1)
+        model.load_state_dict(torch.load(checkpoint_path))
+        if verbose >= 2:
+            print("Loaded weights from {}".format(checkpoint_path))
+
+def fit_model(model, loss_op, optim_op, train_gen, val_gen, epochs,
+              checkpoint_path, patience):
+    """ Analog to Keras fit_generator function.
+
+    # Arguments:
+        model: Model to be finetuned.
+        loss_op: loss operation (BCEWithLogitsLoss or CrossEntropy for e.g.)
+        optim_op: optimization operation (Adam e.g.)
+        train_gen: Training data iterator (DataLoader)
+        val_gen: Validation data iterator (DataLoader)
+        epochs: Number of epochs.
+        checkpoint_path: Filepath where weights will be checkpointed to
+            during training. This file will be rewritten by the function.
+        patience: Patience for callback methods.
+        verbose: Verbosity flag.
+
+    # Returns:
+        Accuracy of the trained model, ONLY if 'evaluate' is set.
+    """
+    # Save original checkpoint
+    torch.save(model.state_dict(), checkpoint_path)
+
+    model.eval()
+    best_loss = np.mean([loss_op(model(Variable(xv)).squeeze(), Variable(yv.float()).squeeze()).data.cpu().numpy()[0] for xv, yv in val_gen])
+    print("original val loss", best_loss)
+
+    epoch_without_impr = 0
+    for epoch in range(epochs):
+        for i, data in enumerate(train_gen):
+            X_train, y_train = data
+            X_train = Variable(X_train, requires_grad=False)
+            y_train = Variable(y_train, requires_grad=False)
+            if torch.cuda.is_available():
+                X_train = X_train.cuda()
+                y_train = y_train.cuda()
+            model.train()
+            optim_op.zero_grad()
+            output = model(X_train)
+            loss = loss_op(output, y_train.float())
+            loss.backward()
+            clip_grad_norm(model.parameters(), 1)
+            optim_op.step()
+
+            acc = evaluate_using_acc(model, [(X_train.data, y_train.data)])
+            print("== Epoch", epoch, "step", i, "train loss", loss.data.cpu().numpy()[0], "train acc", acc)
+
+        model.eval()
+        acc = evaluate_using_acc(model, val_gen)
+        print("val acc", acc)
+
+        val_loss = np.mean([loss_op(model(Variable(xv)).squeeze(), Variable(yv.float()).squeeze()).data.cpu().numpy()[0] for xv, yv in val_gen])
+        print("val loss", val_loss)
+        if best_loss is not None and val_loss >= best_loss:
+            epoch_without_impr += 1
+            print('No improvement over previous best loss: ', best_loss)
+
+        # Save checkpoint
+        if best_loss is None or val_loss < best_loss:
+            best_loss = val_loss
+            torch.save(model.state_dict(), checkpoint_path)
+            print('Saving model at', checkpoint_path)
+
+        # Early stopping
+        if epoch_without_impr >= patience:
+            break
+
+def get_data_loader(X_in, y_in, batch_size, extended_batch_sampler=True, epoch_size=25000, upsample=False, seed=42):
+    """ Returns a dataloader that enables larger epochs on small datasets and
+        has upsampling functionality.
+
+    # Arguments:
+        X_in: Inputs of the given dataset.
+        y_in: Outputs of the given dataset.
+        batch_size: Batch size.
+        epoch_size: Number of samples in an epoch.
+        upsample: Whether upsampling should be done. This flag should only be
+            set on binary class problems.
+
+    # Returns:
+        DataLoader.
+    """
+    dataset = DeepMojiDataset(X_in, y_in)
+
+    if extended_batch_sampler:
+        batch_sampler = DeepMojiBatchSampler(y_in, batch_size, epoch_size=epoch_size, upsample=upsample, seed=seed)
+    else:
+        batch_sampler = BatchSampler(SequentialSampler(y_in), batch_size, drop_last=False)
+
+    return DataLoader(dataset, batch_sampler=batch_sampler, num_workers=0)
+
+class DeepMojiDataset(Dataset):
+    """ A simple Dataset class.
+
+    # Arguments:
+        X_in: Inputs of the given dataset.
+        y_in: Outputs of the given dataset.
+    
+    # __getitem__ output:
+        (torch.LongTensor, torch.LongTensor)
+    """
+    def __init__(self, X_in, y_in):
+        # Check if we have Torch.LongTensor inputs (assume Numpy array otherwise)
+        if not isinstance(X_in, torch.LongTensor):
+            X_in = torch.from_numpy(X_in.astype('int64')).long()
+        if not isinstance(y_in, torch.LongTensor):
+            y_in = torch.from_numpy(y_in.astype('int64')).long()
+
+        self.X_in = torch.split(X_in, 1, dim=0)
+        self.y_in = torch.split(y_in, 1, dim=0)
+
+    def __len__(self):
+        return len(self.X_in)
+
+    def __getitem__(self, idx):
+        return self.X_in[idx].squeeze(), self.y_in[idx].squeeze()
+
+class DeepMojiBatchSampler(object):
+    """A Batch sampler that enables larger epochs on small datasets and
+        has upsampling functionality.
+
+    # Arguments:
+        y_in: Labels of the dataset.
+        batch_size: Batch size.
+        epoch_size: Number of samples in an epoch.
+        upsample: Whether upsampling should be done. This flag should only be
+            set on binary class problems.
+        seed: Random number generator seed.
+
+    # __iter__ output:
+        iterator of lists (batches) of indices in the dataset
+    """
+
+    def __init__(self, y_in, batch_size, epoch_size, upsample, seed):
+        self.batch_size = batch_size
+        self.epoch_size = epoch_size
+        self.upsample = upsample
+
+        np.random.seed(seed)
+
+        if upsample:
+            # Should only be used on binary class problems
+            assert len(y_in.shape) == 1
+            neg = np.where(y_in.numpy() == 0)[0]
+            pos = np.where(y_in.numpy() == 1)[0]
+            assert epoch_size % 2 == 0
+            samples_pr_class = int(epoch_size / 2)
+        else:
+            ind = range(len(y_in))
+
+        if not upsample:
+            # Randomly sample observations in a balanced way
+            self.sample_ind = np.random.choice(ind, epoch_size, replace=True)
+        else:
+            # Randomly sample observations in a balanced way
+            sample_neg = np.random.choice(neg, samples_pr_class, replace=True)
+            sample_pos = np.random.choice(pos, samples_pr_class, replace=True)
+            concat_ind = np.concatenate((sample_neg, sample_pos), axis=0)
+
+            # Shuffle to avoid labels being in specific order
+            # (all negative then positive)
+            p = np.random.permutation(len(concat_ind))
+            self.sample_ind = concat_ind[p]
+
+            label_dist = np.mean(y_in.numpy()[self.sample_ind])
+            assert(label_dist > 0.45)
+            assert(label_dist < 0.55)
+
+    def __iter__(self):
+        # Hand-off data using batch_size
+        for i in range(int(self.epoch_size/self.batch_size)):
+            start = i * self.batch_size
+            end = min(start + self.batch_size, self.epoch_size)
+            yield self.sample_ind[start:end]
+
+    def __len__(self):
+        # Take care of the last (maybe incomplete) batch
+        return (self.epoch_size + self.batch_size - 1) // self.batch_size