# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     name: python3
# ---

# + [markdown] id="view-in-github" colab_type="text"
# <a href="https://colab.research.google.com/github/clayajohnson/gpt-2/blob/interactive-feature/Finetune_GPT_2_on_JRE_transcripts.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>

# + [markdown] id="hd8JsUcWmdf8" colab_type="text"
# # Finetune GPT-2 on Joe Rogan Experience transcripts
#
# By [<NAME>](https://www.linkedin.com/in/clayajohnson/)
#
# *Last updated: July 17th, 2020*
#
# This colab notebook finetunes OpenAi's gpt-2 on transcripts from the Joe Rogan Experience podcast as part of the dialogue modelling transformer network project.
#
# For more info you can visit [DMTNet](https://github.com/clayajohnson/dialogue_modelling_transformer_network).
#
# Begin by copying this notebook to your Google Drive to keep it and save your changes. (File -> Save a Copy in Drive)

# + [markdown] id="a1vsRh7zWRVM" colab_type="text"
# ## Setup
#
# In order to retrain the model, the workspace needs to be configured correctly with the appropriate resources and libraries.
#
# Steps:
# 1. Clone my repo into the colab notebook
# 2. Download the requirements
# 3. Configure project compatible version of tensorflow and cuda support

# + id="riVEKJ_MZ5hX" colab_type="code" colab={}
# Clone the repo with modified training and running scripts
# !git clone -b interactive-feature https://github.com/clayajohnson/gpt-2.git

# + id="ECSOrg_9RU2n" colab_type="code" colab={}
# Move into gpt-2 folder
# %cd /content/gpt-2

# + id="mBOpJTTo_aU5" colab_type="code" colab={}
# Download contents of requirements.txt
# !pip3 install -r requirements.txt

# + id="xapT_JtlRie2" colab_type="code" colab={}
# Install the compatible version of tensorflow with GPU support
# !pip install tensorflow-gpu==1.15.0
# !pip install 'tensorflow-estimator<1.15.0rc0,>=1.14.0rc0' --force-reinstall

# + id="ZpnEZMLRSMyx" colab_type="code" colab={}
# Install compatible cuda support for project
# !wget https://developer.nvidia.com/compute/cuda/9.0/Prod/local_installers/cuda-repo-ubuntu1604-9-0-local_9.0.176-1_amd64-deb
# !dpkg -i cuda-repo-ubuntu1604-9-0-local_9.0.176-1_amd64-deb
# !apt-key add /var/cuda-repo-*/7fa2af80.pub
# !apt-get update
# !apt-get install cuda-9-0
# !export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-9.0/lib64/

# + id="OrHykbUqCu9O" colab_type="code" colab={}
# Move into gpt-2 folder after restarting runtime
# %cd /content/gpt-2

# + [markdown] id="3L96aP_5Xkv6" colab_type="text"
# ## Finetuning
#
# First, a model must be downloaded that will then be finetuned. This notebook uses the 345M parameter model but a smaller 117M parameter model is also available.
#
# Steps:
# 1. Download the model
# 2. Run the training script `train.py` for approximately 70 steps on the JRE dataset (probably need to adjust values for different datasets)
# 3. Run the interactive script `interactive_conditional_samples.py` to test the conversational functionality of the model.

# + id="P-l4vFusbkzS" colab_type="code" colab={}
# Download the 345 million parameter model
# !python3 download_model.py 345M

# + id="Qf2m80RLcYGS" colab_type="code" colab={}
# Run the training script on the JRE dataset (approx. 70 steps yields good results) go to "Runtime > Interupt execution" to stop training.
# !PYTHONPATH=src ./train.py --dataset jre_dataset.txt --model_name 345M --sample_every 10 --learning_rate 0.0001 --restore_from fresh

# + id="iFImXoKrhBoA" colab_type="code" colab={}
# Run the interactive script on the retrained model
# !PYTHONPATH=src ./src/interactive_conditional_samples.py --run_name run1

# + [markdown] id="5zAUfqH3Yuxz" colab_type="text"
# ## Saving the model
#
# Once the model is suitably finetuned, save it to your gdrive. This is done by first mounting your google drive and then copying the checkpoint/run1 folder into a folder `/checkpoints` in your google drive. From here, the folder can be zipped and downloaded onto your local machine.

# + id="Xwty9bEtQizM" colab_type="code" colab={}
# Once you have tested the retrained model and are happy with the results, save the model to your google drive
from google.colab import drive
drive.mount('/content/gdrive')
# %cp -r ./checkpoint/run1/ /content/gdrive/My\ Drive/checkpoints

# + [markdown] id="j5yE1Xa4Zm0w" colab_type="text"
# # LICENSE
#
# MIT License
#
# Copyright (c) 2020 <NAME>
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all
# copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.

StarcoderdataJupyterScriptsDedupFiltered