from sklearn.feature_extraction.text import CountVectorizer

vectorizer = CountVectorizer(min_df=1)

vectorizer.fit([
    "The cat sat on the mat.",
])
vectorizer.vocabulary_

"""
Explanation: Large Scale Text Classification for Sentiment Analysis
Scalability Issues
The sklearn.feature_extraction.text.CountVectorizer and sklearn.feature_extraction.text.TfidfVectorizer classes suffer from a number of scalability issues that all stem from the internal usage of the vocabulary_ attribute (a Python dictionary) used to map the unicode string feature names to the integer feature indices.
The main scalability issues are:

Memory usage of the text vectorizer: the all the string representations of the features are loaded in memory
Parallelization problems for text feature extraction: the vocabulary_ would be a shared state: complex synchronization and overhead
Impossibility to do online or out-of-core / streaming learning: the vocabulary_ needs to be learned from the data: its size cannot be known before making one pass over the full dataset

To better understand the issue let's have a look at how the vocabulary_ attribute work. At fit time the tokens of the corpus are uniquely indentified by a integer index and this mapping stored in the vocabulary:
End of explanation
"""


X = vectorizer.transform([
    "The cat sat on the mat.",
    "This cat is a nice cat.",
]).toarray()

print(len(vectorizer.vocabulary_))
print(vectorizer.get_feature_names())
print(X)

"""
Explanation: The vocabulary is used at transform time to build the occurrence matrix:
End of explanation
"""


vectorizer = CountVectorizer(min_df=1)

vectorizer.fit([
    "The cat sat on the mat.",
    "The quick brown fox jumps over the lazy dog.",
])
vectorizer.vocabulary_

"""
Explanation: Let's refit with a slightly larger corpus:
End of explanation
"""


X = vectorizer.transform([
    "The cat sat on the mat.",
    "This cat is a nice cat.",
]).toarray()

print(len(vectorizer.vocabulary_))
print(vectorizer.get_feature_names())
print(X)

"""
Explanation: The vocabulary_ is the (logarithmically) growing with the size of the training corpus. Note that we could not have built the vocabularies in parallel on the 2 text documents as they share some words hence would require some kind of shared datastructure or synchronization barrier which is complicated to setup, especially if we want to distribute the processing on a cluster.
With this new vocabulary, the dimensionality of the output space is now larger:
End of explanation
"""


import os

sentiment140_folder = os.path.join('datasets', 'sentiment140')
training_csv_file = os.path.join(sentiment140_folder, 'training.1600000.processed.noemoticon.csv')
testing_csv_file = os.path.join(sentiment140_folder, 'testdata.manual.2009.06.14.csv')

"""
Explanation: The Sentiment 140 Dataset
To illustrate the scalability issues of the vocabulary-based vectorizers, let's load a more realistic dataset for a classical text classification task: sentiment analysis on tweets. The goal is to tell apart negative from positive tweets on a variety of topics.
Assuming that the ../fetch_data.py script was run successfully the following files should be available:
End of explanation
"""


!ls -lh datasets/sentiment140/training.1600000.processed.noemoticon.csv

"""
Explanation: Those files were downloaded from the research archive of the http://www.sentiment140.com/ project. The first file was gathered using the twitter streaming API by running stream queries for the positive ":)" and negative ":(" emoticons to collect tweets that are explicitly positive or negative. To make the classification problem non-trivial, the emoticons were stripped out of the text in the CSV files:
End of explanation
"""


FIELDNAMES = ('polarity', 'id', 'date', 'query', 'author', 'text')

def read_csv(csv_file, fieldnames=FIELDNAMES, max_count=None,
             n_partitions=1, partition_id=0):
    
    import csv  # put the import inside for use in IPython.parallel
    def file_opener(csv_file):
        try:
            open(csv_file, 'r', encoding="latin1").close()
            return open(csv_file, 'r', encoding="latin1")
        except TypeError:
            # Python 2 does not have encoding arg
            return open(csv_file, 'rb')
    
    texts = []
    targets = []
    with file_opener(csv_file) as f:
        reader = csv.DictReader(f, fieldnames=fieldnames,
                                delimiter=',', quotechar='"')
        pos_count, neg_count = 0, 0
        for i, d in enumerate(reader):
            if i % n_partitions != partition_id:
                # Skip entry if not in the requested partition
                continue

            if d['polarity'] == '4':
                if max_count and pos_count >= max_count / 2:
                    continue
                pos_count += 1
                texts.append(d['text'])
                targets.append(1)

            elif d['polarity'] == '0':
                if max_count and neg_count >= max_count / 2:
                    continue
                neg_count += 1
                texts.append(d['text'])
                targets.append(-1)

    return texts, targets

%time text_train_all, target_train_all = read_csv(training_csv_file, max_count=200000)

len(text_train_all), len(target_train_all)

"""
Explanation: Let's parse the CSV files and load everything in memory. As loading everything can take up to 2GB, let's limit the collection to 100K tweets of each (positive and negative) out of the total of 1.6M tweets.
End of explanation
"""


for text in text_train_all[:3]:
    print(text + "\n")

print(target_train_all[:3])

"""
Explanation: Let's display the first samples:
End of explanation
"""


for text in text_train_all[-3:]:
    print(text + "\n")

print(target_train_all[-3:])

"""
Explanation: A polarity of "0" means negative while a polarity of "4" means positive. All the positive tweets are at the end of the file:
End of explanation
"""


from sklearn.cross_validation import train_test_split

text_train_small, text_validation, target_train_small, target_validation = train_test_split(
    text_train_all, np.array(target_train_all), test_size=.5, random_state=42)

len(text_train_small)

(target_train_small == -1).sum(), (target_train_small == 1).sum()

len(text_validation)

(target_validation == -1).sum(), (target_validation == 1).sum()

"""
Explanation: Let's split the training CSV file into a smaller training set and a validation set with 100k random tweets each:
End of explanation
"""


text_test_all, target_test_all = read_csv(testing_csv_file)

len(text_test_all), len(target_test_all)

"""
Explanation: Let's open the manually annotated tweet files. The evaluation set also has neutral tweets with a polarity of "2" which we ignore. We can build the final evaluation set with only the positive and negative tweets of the evaluation CSV file:
End of explanation
"""


from sklearn.utils.murmurhash import murmurhash3_bytes_u32

# encode for python 3 compatibility
for word in "the cat sat on the mat".encode("utf-8").split():
    print("{0} => {1}".format(
        word, murmurhash3_bytes_u32(word, 0) % 2 ** 20))

"""
Explanation: The Hashing Trick
Remember the bag of word representation using a vocabulary based vectorizer:

To workaround the limitations of the vocabulary-based vectorizers, one can use the hashing trick. Instead of building and storing an explicit mapping from the feature names to the feature indices in a Python dict, we can just use a hash function and a modulus operation:
End of explanation
"""


from sklearn.feature_extraction.text import HashingVectorizer

h_vectorizer = HashingVectorizer(encoding='latin-1')
h_vectorizer

"""
Explanation: This mapping is completely stateless and the dimensionality of the output space is explicitly fixed in advance (here we use a modulo 2 ** 20 which means roughly 1M dimensions). The makes it possible to workaround the limitations of the vocabulary based vectorizer both for parallelizability and online / out-of-core learning.
The HashingVectorizer class is an alternative to the TfidfVectorizer class with use_idf=False that internally uses the murmurhash hash function:
End of explanation
"""


analyzer = h_vectorizer.build_analyzer()
analyzer('This is a test sentence.')

"""
Explanation: It shares the same "preprocessor", "tokenizer" and "analyzer" infrastructure:
End of explanation
"""


%time X_train_small = h_vectorizer.transform(text_train_small)

"""
Explanation: We can vectorize our datasets into a scipy sparse matrix exactly as we would have done with the CountVectorizer or TfidfVectorizer, except that we can directly call the transform method: there is no need to fit as HashingVectorizer is a stateless transformer:
End of explanation
"""


X_train_small

"""
Explanation: The dimension of the output is fixed ahead of time to n_features=2 ** 20 by default (nearly 1M features) to minimize the rate of collision on most classification problem while having reasonably sized linear models (1M weights in the coef_ attribute):
End of explanation
"""


from sklearn.linear_model import PassiveAggressiveClassifier
from sklearn.pipeline import Pipeline

h_pipeline = Pipeline((
    ('vec', HashingVectorizer(encoding='latin-1')),
    ('clf', PassiveAggressiveClassifier(C=1, n_iter=1)),
))

%time h_pipeline.fit(text_train_small, target_train_small).score(text_validation, target_validation)

"""
Explanation: As only the non-zero elements are stored, n_features has little impact on the actual size of the data in memory. We can combine the hashing vectorizer with a Passive-Aggressive linear model in a pipeline:
End of explanation
"""


h_pipeline.score(text_test_all, target_test_all)

"""
Explanation: Let's check that the score on the validation set is reasonably in line with the set of manually annotated tweets:
End of explanation
"""


from sklearn.feature_extraction.text import TfidfVectorizer

vocabulary_vec = TfidfVectorizer(encoding='latin-1', use_idf=False)
vocabulary_pipeline = Pipeline((
    ('vec', vocabulary_vec),
    ('clf', PassiveAggressiveClassifier(C=1, n_iter=1)),
))

%time vocabulary_pipeline.fit(text_train_small, target_train_small).score(text_validation, target_validation)

"""
Explanation: As the text_train_small dataset is not that big we can still use a vocabulary based vectorizer to check that the hashing collisions are not causing any significant performance drop on the validation set (WARNING this is twice as slow as the hashing vectorizer version, skip this cell if your computer is too slow):
End of explanation
"""


len(vocabulary_vec.vocabulary_)

"""
Explanation: We get almost the same score but almost twice as slower with also a big, slow to (un)pickle datastructure in memory:
End of explanation
"""


from random import Random


class InfiniteStreamGenerator(object):
    """Simulate random polarity queries on the twitter streaming API"""
    
    def __init__(self, texts, targets, seed=0, batchsize=100):
        self.texts_pos = [text for text, target in zip(texts, targets)
                               if target > 0]
        self.texts_neg = [text for text, target in zip(texts, targets)
                               if target <= 0]
        self.rng = Random(seed)
        self.batchsize = batchsize

    def next_batch(self, batchsize=None):
        batchsize = self.batchsize if batchsize is None else batchsize
        texts, targets = [], []
        for i in range(batchsize):
            # Select the polarity randomly
            target = self.rng.choice((-1, 1))
            targets.append(target)
            
            # Combine 2 random texts of the right polarity
            pool = self.texts_pos if target > 0 else self.texts_neg
            text = self.rng.choice(pool) + " " + self.rng.choice(pool)
            texts.append(text)
        return texts, targets

infinite_stream = InfiniteStreamGenerator(text_train_small, target_train_small)

texts_in_batch, targets_in_batch = infinite_stream.next_batch(batchsize=3)

for t in texts_in_batch:
    print(t + "\n")

targets_in_batch

"""
Explanation: More info and reference for the original papers on the Hashing Trick in the answers to this http://metaoptimize.com/qa question: What is the Hashing Trick?.
Out-of-Core learning
Out-of-Core learning is the task of training a machine learning model on a dataset that does not fit in the main memory. This requires the following conditions:

a feature extraction layer with fixed output dimensionality
knowing the list of all classes in advance (in this case we only have positive and negative tweets)
a machine learning algorithm that supports incremental learning (the partial_fit method in scikit-learn).

Let us simulate an infinite tweeter stream that can generate batches of annotated tweet texts and there polarity. We can do this by recombining randomly pairs of positive or negative tweets from our fixed dataset:
End of explanation
"""


n_batches = 1000
validation_scores = []
training_set_size = []

# Build the vectorizer and the classifier
h_vectorizer = HashingVectorizer(encoding='latin-1')
clf = PassiveAggressiveClassifier(C=1)

# Extract the features for the validation once and for all
X_validation = h_vectorizer.transform(text_validation)
classes = np.array([-1, 1])

n_samples = 0
for i in range(n_batches):
    
    texts_in_batch, targets_in_batch = infinite_stream.next_batch()    
    n_samples += len(texts_in_batch)

    # Vectorize the text documents in the batch
    X_batch = h_vectorizer.transform(texts_in_batch)
    
    # Incrementally train the model on the new batch
    clf.partial_fit(X_batch, targets_in_batch, classes=classes)
    
    if n_samples % 100 == 0:
        # Compute the validation score of the current state of the model
        score = clf.score(X_validation, target_validation)
        validation_scores.append(score)
        training_set_size.append(n_samples)

    if i % 100 == 0:
        print("n_samples: {0}, score: {1:.4f}".format(n_samples, score))

"""
Explanation: We can now use our infinte tweet source to train an online machine learning algorithm using the hashing vectorizer. Note the use of the partial_fit method of the PassiveAggressiveClassifier instance in place of the traditional call to the fit method that needs access to the full training set.
From time to time, we evaluate the current predictive performance of the model on our validation set that is guaranteed to not overlap with the infinite training set source:
End of explanation
"""


plt.plot(training_set_size, validation_scores)
plt.ylim(0.5, 1)
plt.xlabel("Number of samples")
plt.ylabel("Validation score")

"""
Explanation: We can now plot the collected validation score values, versus the number of samples generated by the infinite source and feed to the model:
End of explanation
"""


