# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# # Word2Vec

# +
# implemented in this Google Colab
# https://colab.research.google.com/drive/1CyKR0zDM4_x6qbKI_xBpqIUKlh11RH_s?usp=sharing
# -

# In order to decide whether to train our own embeddings or use pre-trained embeddings, a good rule of thumb is to compute the vocabulary overlap. If the overlap between the vocabulary of our custom domain and that of pre-trained word embeddings is greater than 80%, pre-trained word embeddings tend to give good results in text classification.
#
# An important factor to consider when deploying models with embedding-based feature extraction approaches is that the learned or pre-trained embedding models have to be stored and loaded into memory while using these approaches. If the model itself is bulky (e.g., the pre-trained model we used takes 3.6 GB), we need to factor this into our deployment needs.

# # Subword Embeddings and fastText
#
# It has two advantages:
# - This approach can handle words that did not appear in training data (OOV)
# - The implementation facilitates extremely fast learning on even very large corpora.

# +
# implemented in this Google Colab
# https://colab.research.google.com/drive/1CyKR0zDM4_x6qbKI_xBpqIUKlh11RH_s?usp=sharing
# -

# fastText is extremely fast to train and very useful for setting up strong baselines. The downside is the model size.
#
# To handle this problem, fastText comes with options to reduce the memory footprint of its classification models with minimal reduction in classification performance. It does this by doing vocabulary pruning and using compression algorithms.

# # Doc2vec

# +
# implemented in this Google Colab
# https://colab.research.google.com/drive/1CyKR0zDM4_x6qbKI_xBpqIUKlh11RH_s?usp=sharing
# -

# An important point to keep in mind when using Doc2vec is the same as for fastText: if we have to use Doc2vec for feature representation, we have to store the model that learned the representation. While it's not typically as bulky as fastText, it’s also not as fast to train. Such trade-offs need to be considered and compared before we make a deployment decision.



StarcoderdataJupyterScriptsDedupFiltered