examples/fine-tune-modernbert-classifier.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Finetune a text classifier with ModernBERT and synthetic data from an LLM\n",
    "\n",
    "LLMs are great general purpose models, but they are not always the best choice for a specific task. Therefore, smaller and more specialized models are important for sustainable, efficient, and cheaper AI.\n",
    "\n",
    "A lack of dedicated datasets is a common problem for smaller and more specialized models. This is because it is difficult to find a dataset that is both representative and diverse enough for a specific task. We solve this problem by generating a synthetic dataset from an LLM using the `synthetic-data-generator`, which is available as [Hugging Face Space](https://huggingface.co/spaces/argilla/synthetic-data-generator) or on [GitHub](https://github.com/argilla-io/synthetic-data-generator).\n",
    "\n",
    "In this example, we will finetune a ModernBERT model on a synthetic dataset generated from the synthetic-data-generator. Showing the effectiveness of synthetic data and the novel ModernBERT model, which is new and improved version of BERT models, with 8192 token context length, significantly better downstream performance, and much faster processing speeds.\n",
    "\n",
    "## Install the dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
      "To disable this warning, you can either:\n",
      "\t- Avoid using `tokenizers` before the fork if possible\n",
      "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting torch==2.5.0\n",
      "  Downloading torch-2.5.0-cp311-none-macosx_11_0_arm64.whl.metadata (28 kB)\n",
      "Collecting torchvision==0.20.0\n",
      "  Using cached torchvision-0.20.0-cp311-cp311-macosx_11_0_arm64.whl.metadata (6.1 kB)\n",
      "Requirement already satisfied: tensorboard in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (2.18.0)\n",
      "Requirement already satisfied: filelock in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from torch==2.5.0) (3.16.1)\n",
      "Requirement already satisfied: typing-extensions>=4.8.0 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from torch==2.5.0) (4.12.2)\n",
      "Requirement already satisfied: networkx in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from torch==2.5.0) (3.4.2)\n",
      "Requirement already satisfied: jinja2 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from torch==2.5.0) (3.1.4)\n",
      "Requirement already satisfied: fsspec in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from torch==2.5.0) (2024.9.0)\n",
      "Requirement already satisfied: sympy==1.13.1 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from torch==2.5.0) (1.13.1)\n",
      "Requirement already satisfied: numpy in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from torchvision==0.20.0) (2.0.2)\n",
      "Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from torchvision==0.20.0) (11.0.0)\n",
      "Requirement already satisfied: mpmath<1.4,>=1.1.0 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from sympy==1.13.1->torch==2.5.0) (1.3.0)\n",
      "Requirement already satisfied: absl-py>=0.4 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from tensorboard) (2.1.0)\n",
      "Requirement already satisfied: grpcio>=1.48.2 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from tensorboard) (1.68.1)\n",
      "Requirement already satisfied: markdown>=2.6.8 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from tensorboard) (3.7)\n",
      "Requirement already satisfied: packaging in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from tensorboard) (24.2)\n",
      "Requirement already satisfied: protobuf!=4.24.0,>=3.19.6 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from tensorboard) (5.29.2)\n",
      "Requirement already satisfied: setuptools>=41.0.0 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from tensorboard) (75.6.0)\n",
      "Requirement already satisfied: six>1.9 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from tensorboard) (1.17.0)\n",
      "Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from tensorboard) (0.7.2)\n",
      "Requirement already satisfied: werkzeug>=1.0.1 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from tensorboard) (3.1.3)\n",
      "Requirement already satisfied: MarkupSafe>=2.1.1 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from werkzeug>=1.0.1->tensorboard) (2.1.5)\n",
      "Downloading torch-2.5.0-cp311-none-macosx_11_0_arm64.whl (64.3 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m64.3/64.3 MB\u001b[0m \u001b[31m8.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0mm\n",
      "\u001b[?25hDownloading torchvision-0.20.0-cp311-cp311-macosx_11_0_arm64.whl (1.8 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.8/1.8 MB\u001b[0m \u001b[31m33.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hInstalling collected packages: torch, torchvision\n",
      "  Attempting uninstall: torch\n",
      "    Found existing installation: torch 2.4.1\n",
      "    Uninstalling torch-2.4.1:\n",
      "      Successfully uninstalled torch-2.4.1\n",
      "  Attempting uninstall: torchvision\n",
      "    Found existing installation: torchvision 0.19.1\n",
      "    Uninstalling torchvision-0.19.1:\n",
      "      Successfully uninstalled torchvision-0.19.1\n",
      "Successfully installed torch-2.5.0 torchvision-0.20.0\n",
      "\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.1.2\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.3.1\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
      "To disable this warning, you can either:\n",
      "\t- Avoid using `tokenizers` before the fork if possible\n",
      "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting flash-attn\n",
      "  Using cached flash_attn-2.7.2.post1.tar.gz (3.1 MB)\n",
      "  Installing build dependencies ... \u001b[?25ldone\n",
      "\u001b[?25h  Getting requirements to build wheel ... \u001b[?25lerror\n",
      "  \u001b[1;31merror\u001b[0m: \u001b[1msubprocess-exited-with-error\u001b[0m\n",
      "  \n",
      "  \u001b[31m×\u001b[0m \u001b[32mGetting requirements to build wheel\u001b[0m did not run successfully.\n",
      "  \u001b[31m│\u001b[0m exit code: \u001b[1;36m1\u001b[0m\n",
      "  \u001b[31m╰─>\u001b[0m \u001b[31m[20 lines of output]\u001b[0m\n",
      "  \u001b[31m   \u001b[0m Traceback (most recent call last):\n",
      "  \u001b[31m   \u001b[0m   File \"/Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py\", line 353, in <module>\n",
      "  \u001b[31m   \u001b[0m     main()\n",
      "  \u001b[31m   \u001b[0m   File \"/Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py\", line 335, in main\n",
      "  \u001b[31m   \u001b[0m     json_out['return_val'] = hook(**hook_input['kwargs'])\n",
      "  \u001b[31m   \u001b[0m                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
      "  \u001b[31m   \u001b[0m   File \"/Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py\", line 118, in get_requires_for_build_wheel\n",
      "  \u001b[31m   \u001b[0m     return hook(config_settings)\n",
      "  \u001b[31m   \u001b[0m            ^^^^^^^^^^^^^^^^^^^^^\n",
      "  \u001b[31m   \u001b[0m   File \"/private/var/folders/9t/msy700h16jz3q35qvg4z1ln40000gn/T/pip-build-env-65l3ta85/overlay/lib/python3.11/site-packages/setuptools/build_meta.py\", line 334, in get_requires_for_build_wheel\n",
      "  \u001b[31m   \u001b[0m     return self._get_build_requires(config_settings, requirements=[])\n",
      "  \u001b[31m   \u001b[0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
      "  \u001b[31m   \u001b[0m   File \"/private/var/folders/9t/msy700h16jz3q35qvg4z1ln40000gn/T/pip-build-env-65l3ta85/overlay/lib/python3.11/site-packages/setuptools/build_meta.py\", line 304, in _get_build_requires\n",
      "  \u001b[31m   \u001b[0m     self.run_setup()\n",
      "  \u001b[31m   \u001b[0m   File \"/private/var/folders/9t/msy700h16jz3q35qvg4z1ln40000gn/T/pip-build-env-65l3ta85/overlay/lib/python3.11/site-packages/setuptools/build_meta.py\", line 522, in run_setup\n",
      "  \u001b[31m   \u001b[0m     super().run_setup(setup_script=setup_script)\n",
      "  \u001b[31m   \u001b[0m   File \"/private/var/folders/9t/msy700h16jz3q35qvg4z1ln40000gn/T/pip-build-env-65l3ta85/overlay/lib/python3.11/site-packages/setuptools/build_meta.py\", line 320, in run_setup\n",
      "  \u001b[31m   \u001b[0m     exec(code, locals())\n",
      "  \u001b[31m   \u001b[0m   File \"<string>\", line 21, in <module>\n",
      "  \u001b[31m   \u001b[0m ModuleNotFoundError: No module named 'torch'\n",
      "  \u001b[31m   \u001b[0m \u001b[31m[end of output]\u001b[0m\n",
      "  \n",
      "  \u001b[1;35mnote\u001b[0m: This error originates from a subprocess, and is likely not a problem with pip.\n",
      "\u001b[?25h\u001b[1;31merror\u001b[0m: \u001b[1msubprocess-exited-with-error\u001b[0m\n",
      "\n",
      "\u001b[31m×\u001b[0m \u001b[32mGetting requirements to build wheel\u001b[0m did not run successfully.\n",
      "\u001b[31m│\u001b[0m exit code: \u001b[1;36m1\u001b[0m\n",
      "\u001b[31m╰─>\u001b[0m See above for output.\n",
      "\n",
      "\u001b[1;35mnote\u001b[0m: This error originates from a subprocess, and is likely not a problem with pip.\n",
      "\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.1.2\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.3.1\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
      "To disable this warning, you can either:\n",
      "\t- Avoid using `tokenizers` before the fork if possible\n",
      "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: datasets==3.1.0 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (3.1.0)\n",
      "Requirement already satisfied: accelerate==1.2.1 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (1.2.1)\n",
      "Requirement already satisfied: hf-transfer==0.1.8 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (0.1.8)\n",
      "Requirement already satisfied: filelock in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from datasets==3.1.0) (3.16.1)\n",
      "Requirement already satisfied: numpy>=1.17 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from datasets==3.1.0) (2.0.2)\n",
      "Requirement already satisfied: pyarrow>=15.0.0 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from datasets==3.1.0) (18.1.0)\n",
      "Requirement already satisfied: dill<0.3.9,>=0.3.0 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from datasets==3.1.0) (0.3.8)\n",
      "Requirement already satisfied: pandas in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from datasets==3.1.0) (2.2.3)\n",
      "Requirement already satisfied: requests>=2.32.2 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from datasets==3.1.0) (2.32.3)\n",
      "Requirement already satisfied: tqdm>=4.66.3 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from datasets==3.1.0) (4.67.1)\n",
      "Requirement already satisfied: xxhash in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from datasets==3.1.0) (3.5.0)\n",
      "Requirement already satisfied: multiprocess<0.70.17 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from datasets==3.1.0) (0.70.16)\n",
      "Requirement already satisfied: fsspec<=2024.9.0,>=2023.1.0 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from fsspec[http]<=2024.9.0,>=2023.1.0->datasets==3.1.0) (2024.9.0)\n",
      "Requirement already satisfied: aiohttp in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from datasets==3.1.0) (3.11.10)\n",
      "Requirement already satisfied: huggingface-hub>=0.23.0 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from datasets==3.1.0) (0.27.0)\n",
      "Requirement already satisfied: packaging in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from datasets==3.1.0) (24.2)\n",
      "Requirement already satisfied: pyyaml>=5.1 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from datasets==3.1.0) (6.0.2)\n",
      "Requirement already satisfied: psutil in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from accelerate==1.2.1) (6.1.1)\n",
      "Requirement already satisfied: torch>=1.10.0 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from accelerate==1.2.1) (2.5.0)\n",
      "Requirement already satisfied: safetensors>=0.4.3 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from accelerate==1.2.1) (0.4.5)\n",
      "Requirement already satisfied: aiohappyeyeballs>=2.3.0 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from aiohttp->datasets==3.1.0) (2.4.4)\n",
      "Requirement already satisfied: aiosignal>=1.1.2 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from aiohttp->datasets==3.1.0) (1.3.2)\n",
      "Requirement already satisfied: attrs>=17.3.0 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from aiohttp->datasets==3.1.0) (24.3.0)\n",
      "Requirement already satisfied: frozenlist>=1.1.1 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from aiohttp->datasets==3.1.0) (1.5.0)\n",
      "Requirement already satisfied: multidict<7.0,>=4.5 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from aiohttp->datasets==3.1.0) (6.1.0)\n",
      "Requirement already satisfied: propcache>=0.2.0 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from aiohttp->datasets==3.1.0) (0.2.1)\n",
      "Requirement already satisfied: yarl<2.0,>=1.17.0 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from aiohttp->datasets==3.1.0) (1.18.3)\n",
      "Requirement already satisfied: typing-extensions>=3.7.4.3 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from huggingface-hub>=0.23.0->datasets==3.1.0) (4.12.2)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from requests>=2.32.2->datasets==3.1.0) (3.4.0)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from requests>=2.32.2->datasets==3.1.0) (3.10)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from requests>=2.32.2->datasets==3.1.0) (2.2.3)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from requests>=2.32.2->datasets==3.1.0) (2024.12.14)\n",
      "Requirement already satisfied: networkx in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from torch>=1.10.0->accelerate==1.2.1) (3.4.2)\n",
      "Requirement already satisfied: jinja2 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from torch>=1.10.0->accelerate==1.2.1) (3.1.4)\n",
      "Requirement already satisfied: sympy==1.13.1 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from torch>=1.10.0->accelerate==1.2.1) (1.13.1)\n",
      "Requirement already satisfied: mpmath<1.4,>=1.1.0 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from sympy==1.13.1->torch>=1.10.0->accelerate==1.2.1) (1.3.0)\n",
      "Requirement already satisfied: python-dateutil>=2.8.2 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from pandas->datasets==3.1.0) (2.9.0.post0)\n",
      "Requirement already satisfied: pytz>=2020.1 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from pandas->datasets==3.1.0) (2024.2)\n",
      "Requirement already satisfied: tzdata>=2022.7 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from pandas->datasets==3.1.0) (2024.2)\n",
      "Requirement already satisfied: six>=1.5 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from python-dateutil>=2.8.2->pandas->datasets==3.1.0) (1.17.0)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from jinja2->torch>=1.10.0->accelerate==1.2.1) (2.1.5)\n",
      "\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.1.2\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.3.1\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
      "To disable this warning, you can either:\n",
      "\t- Avoid using `tokenizers` before the fork if possible\n",
      "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting git+https://github.com/huggingface/transformers.git@6e0515e99c39444caae39472ee1b2fd76ece32f1\n",
      "  Cloning https://github.com/huggingface/transformers.git (to revision 6e0515e99c39444caae39472ee1b2fd76ece32f1) to /private/var/folders/9t/msy700h16jz3q35qvg4z1ln40000gn/T/pip-req-build-eqq5fh8a\n",
      "  Running command git clone --filter=blob:none --quiet https://github.com/huggingface/transformers.git /private/var/folders/9t/msy700h16jz3q35qvg4z1ln40000gn/T/pip-req-build-eqq5fh8a\n",
      "  Running command git rev-parse -q --verify 'sha^6e0515e99c39444caae39472ee1b2fd76ece32f1'\n",
      "  Running command git fetch -q https://github.com/huggingface/transformers.git 6e0515e99c39444caae39472ee1b2fd76ece32f1\n",
      "  Running command git checkout -q 6e0515e99c39444caae39472ee1b2fd76ece32f1\n",
      "  Resolved https://github.com/huggingface/transformers.git to commit 6e0515e99c39444caae39472ee1b2fd76ece32f1\n",
      "  Installing build dependencies ... \u001b[?25ldone\n",
      "\u001b[?25h  Getting requirements to build wheel ... \u001b[?25ldone\n",
      "\u001b[?25h  Preparing metadata (pyproject.toml) ... \u001b[?25ldone\n",
      "\u001b[?25hRequirement already satisfied: filelock in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from transformers==4.48.0.dev0) (3.16.1)\n",
      "Requirement already satisfied: huggingface-hub<1.0,>=0.24.0 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from transformers==4.48.0.dev0) (0.27.0)\n",
      "Requirement already satisfied: numpy>=1.17 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from transformers==4.48.0.dev0) (2.0.2)\n",
      "Requirement already satisfied: packaging>=20.0 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from transformers==4.48.0.dev0) (24.2)\n",
      "Requirement already satisfied: pyyaml>=5.1 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from transformers==4.48.0.dev0) (6.0.2)\n",
      "Requirement already satisfied: regex!=2019.12.17 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from transformers==4.48.0.dev0) (2024.11.6)\n",
      "Requirement already satisfied: requests in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from transformers==4.48.0.dev0) (2.32.3)\n",
      "Requirement already satisfied: tokenizers<0.22,>=0.21 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from transformers==4.48.0.dev0) (0.21.0)\n",
      "Requirement already satisfied: safetensors>=0.4.1 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from transformers==4.48.0.dev0) (0.4.5)\n",
      "Requirement already satisfied: tqdm>=4.27 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from transformers==4.48.0.dev0) (4.67.1)\n",
      "Requirement already satisfied: fsspec>=2023.5.0 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from huggingface-hub<1.0,>=0.24.0->transformers==4.48.0.dev0) (2024.9.0)\n",
      "Requirement already satisfied: typing-extensions>=3.7.4.3 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from huggingface-hub<1.0,>=0.24.0->transformers==4.48.0.dev0) (4.12.2)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from requests->transformers==4.48.0.dev0) (3.4.0)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from requests->transformers==4.48.0.dev0) (3.10)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from requests->transformers==4.48.0.dev0) (2.2.3)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages (from requests->transformers==4.48.0.dev0) (2024.12.14)\n",
      "\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.1.2\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.3.1\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "# Install Pytorch & other libraries\n",
    "%pip install \"torch==2.5.0\" \"torchvision==0.20.0\" \n",
    "%pip install \"setuptools<71.0.0\" scikit-learn \n",
    " \n",
    "# Install Hugging Face libraries\n",
    "%pip install  --upgrade \\\n",
    "  \"datasets==3.1.0\" \\\n",
    "  \"accelerate==1.2.1\" \\\n",
    "  \"hf-transfer==0.1.8\"\n",
    " \n",
    "# ModernBERT is not yet available in an official release, so we need to install it from github\n",
    "%pip install \"git+https://github.com/huggingface/transformers.git@6e0515e99c39444caae39472ee1b2fd76ece32f1\" --upgrade"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The problem\n",
    "\n",
    "The [nvidia/domain-classifier](https://huggingface.co/nvidia/domain-classifier), is a model that can classify the domain of a text which can help with curating data. This model is cool but is based on the Deberta V3 Base, which is an outdated architecture that requires custom code to run, has a context length of 512 tokens, and is not as fast as the ModernBERT model. The labels for the model are:\n",
    "\n",
    "```\n",
    "'Adult', 'Arts_and_Entertainment', 'Autos_and_Vehicles', 'Beauty_and_Fitness', 'Books_and_Literature', 'Business_and_Industrial', 'Computers_and_Electronics', 'Finance', 'Food_and_Drink', 'Games', 'Health', 'Hobbies_and_Leisure', 'Home_and_Garden', 'Internet_and_Telecom', 'Jobs_and_Education', 'Law_and_Government', 'News', 'Online_Communities', 'People_and_Society', 'Pets_and_Animals', 'Real_Estate', 'Science', 'Sensitive_Subjects', 'Shopping', 'Sports', 'Travel_and_Transportation'\n",
    "```\n",
    "\n",
    "The data on which the model was trained is not available, so we cannot use it for our purposes. We can however generate a synthetic data to solve this problem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "source": [
    "## Let's generate some data\n",
    "\n",
    "Let's go to the [hosted Hugging Face Space](https://huggingface.co/spaces/argilla/synthetic-data-generator) to generate the data. This is done in three steps 1) we come up with a dataset description, 2) iterate on the task configuration, and 3) generate and push the data to Hugging Face. The full flow can be found in [this blogpost](https://huggingface.co/blog/synthetic-data-generator). \n",
    "\n",
    "For this example, we will generate 1000 examples with a temperature of 1. After some iteration, we come up with the following system prompt:\n",
    "\n",
    "```\n",
    "Long texts (at least 2000 words) from various media sources like Wikipedia, Reddit, Common Crawl, websites, commercials, online forums, books, newspapers and folders that cover multiple topics. Classify the text based on its main subject matter into one of the following categories\n",
    "```\n",
    "\n",
    "We press the \"Push to Hub\" button and wait for the data to be generated. This takes a few minutes and we end up with a dataset with 1000 examples. The labels are nicely distributed across the categories, varied in length, and the texts look diverse and interesting.\n",
    "\n",
    "<iframe\n",
    "  src=\"https://huggingface.co/datasets/argilla/synthetic-domain-text-classification/embed/viewer/default/train\"\n",
    "  frameborder=\"0\"\n",
    "  width=\"100%\"\n",
    "  height=\"560px\"\n",
    "></iframe>\n",
    "\n",
    "The data is pushed to Argilla to so we recommend inspecting and validating the labels before finetuning the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Finetuning the ModernBERT model\n",
    "\n",
    "We mostly rely on the blog from [Phillip Schmid](https://www.philschmid.de/fine-tune-modern-bert-in-2025). I will basic consumer hardware, my Apple M1 Max with 32GB of shared memory. We will use the `datasets` library to load the data and the `transformers` library to finetune the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'text': 'Recently, there has been an increase in property values within the suburban areas of several cities due to improvements in infrastructure and lifestyle amenities such as parks, retail stores, and educational institutions nearby. Additionally, new housing developments are emerging, catering to different family needs with varying sizes and price ranges. These changes have influenced investment decisions for many looking to buy or sell properties.',\n",
       " 'label': 14}"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from datasets import load_dataset\n",
    "from datasets.arrow_dataset import Dataset\n",
    "from datasets.dataset_dict import DatasetDict, IterableDatasetDict\n",
    "from datasets.iterable_dataset import IterableDataset\n",
    " \n",
    "# Dataset id from huggingface.co/dataset\n",
    "dataset_id = \"argilla/synthetic-domain-text-classification\"\n",
    " \n",
    "# Load raw dataset\n",
    "train_dataset = load_dataset(dataset_id, split='train')\n",
    "\n",
    "split_dataset = train_dataset.train_test_split(test_size=0.1)\n",
    "split_dataset['train'][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we need to tokenize the data. We will use the `AutoTokenizer` class from the `transformers` library to load the tokenizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Map: 100%|██████████| 900/900 [00:00<00:00, 4787.61 examples/s]\n",
      "Map: 100%|██████████| 100/100 [00:00<00:00, 4163.70 examples/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "dict_keys(['labels', 'input_ids', 'attention_mask'])"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from transformers import AutoTokenizer\n",
    " \n",
    "# Model id to load the tokenizer\n",
    "model_id = \"answerdotai/ModernBERT-base\"\n",
    "\n",
    "# Load Tokenizer\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_id)\n",
    " \n",
    "# Tokenize helper function\n",
    "def tokenize(batch):\n",
    "    return tokenizer(batch['text'], padding='max_length', truncation=True, return_tensors=\"pt\")\n",
    " \n",
    "# Tokenize dataset\n",
    "if \"label\" in split_dataset[\"train\"].features.keys():\n",
    "    split_dataset =  split_dataset.rename_column(\"label\", \"labels\") # to match Trainer\n",
    "tokenized_dataset = split_dataset.map(tokenize, batched=True, remove_columns=[\"text\"])\n",
    " \n",
    "tokenized_dataset[\"train\"].features.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we need to prepare the model. We will use the `AutoModelForSequenceClassification` class from the `transformers` library to load the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of ModernBertForSequenceClassification were not initialized from the model checkpoint at answerdotai/ModernBERT-base and are newly initialized: ['classifier.bias', 'classifier.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "from transformers import AutoModelForSequenceClassification\n",
    " \n",
    "# Model id to load the tokenizer\n",
    "model_id = \"answerdotai/ModernBERT-base\"\n",
    " \n",
    "# Prepare model labels - useful for inference\n",
    "labels = tokenized_dataset[\"train\"].features[\"labels\"].names\n",
    "num_labels = len(labels)\n",
    "label2id, id2label = dict(), dict()\n",
    "for i, label in enumerate(labels):\n",
    "    label2id[label] = str(i)\n",
    "    id2label[str(i)] = label\n",
    " \n",
    "# Download the model from huggingface.co/models\n",
    "model = AutoModelForSequenceClassification.from_pretrained(\n",
    "    model_id, num_labels=num_labels, label2id=label2id, id2label=id2label,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use a simple F1 score as the evaluation metric."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.metrics import f1_score\n",
    " \n",
    "# Metric helper method\n",
    "def compute_metrics(eval_pred):\n",
    "    predictions, labels = eval_pred\n",
    "    predictions = np.argmax(predictions, axis=1)\n",
    "    score = f1_score(\n",
    "            labels, predictions, labels=labels, pos_label=1, average=\"weighted\"\n",
    "        )\n",
    "    return {\"f1\": float(score) if score == 1 else score}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we need to define the training arguments. We will use the `TrainingArguments` class from the `transformers` library to define the training arguments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/davidberenstein/Documents/programming/argilla/synthetic-data-generator/.venv/lib/python3.11/site-packages/transformers/training_args.py:2241: UserWarning: `use_mps_device` is deprecated and will be removed in version 5.0 of 🤗 Transformers. `mps` device will be used by default if available similar to the way `cuda` device is used.Therefore, no action from user is required. \n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "from huggingface_hub import HfFolder\n",
    "from transformers import Trainer, TrainingArguments\n",
    " \n",
    "# Define training args\n",
    "training_args = TrainingArguments(\n",
    "    output_dir= \"ModernBERT-domain-classifier\",\n",
    "    per_device_train_batch_size=32,\n",
    "    per_device_eval_batch_size=16,\n",
    "    learning_rate=5e-5,\n",
    "\t\tnum_train_epochs=5,\n",
    "    bf16=True, # bfloat16 training \n",
    "    optim=\"adamw_torch_fused\", # improved optimizer \n",
    "    # logging & evaluation strategies\n",
    "    logging_strategy=\"steps\",\n",
    "    logging_steps=100,\n",
    "    eval_strategy=\"epoch\",\n",
    "    save_strategy=\"epoch\",\n",
    "    save_total_limit=2,\n",
    "    load_best_model_at_end=True,\n",
    "    use_mps_device=True,\n",
    "    metric_for_best_model=\"f1\",\n",
    "    # push to hub parameters\n",
    "    push_to_hub=True,\n",
    "    hub_strategy=\"every_save\",\n",
    "    hub_token=HfFolder.get_token(),\n",
    ")\n",
    " \n",
    "# Create a Trainer instance\n",
    "trainer = Trainer(\n",
    "    model=model,\n",
    "    args=training_args,\n",
    "    train_dataset=tokenized_dataset[\"train\"],\n",
    "    eval_dataset=tokenized_dataset[\"test\"],\n",
    "    compute_metrics=compute_metrics,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                 \n",
      " 20%|██        | 29/145 [11:32<33:16, 17.21s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.729780912399292, 'eval_f1': 0.7743598318036522, 'eval_runtime': 3.5337, 'eval_samples_per_second': 28.299, 'eval_steps_per_second': 1.981, 'epoch': 1.0}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                  \n",
      " 40%|████      | 58/145 [22:57<25:56, 17.89s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.4369044005870819, 'eval_f1': 0.8310764765820946, 'eval_runtime': 3.3266, 'eval_samples_per_second': 30.061, 'eval_steps_per_second': 2.104, 'epoch': 2.0}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                \n",
      " 60%|██████    | 87/145 [35:16<17:06, 17.70s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.6091340184211731, 'eval_f1': 0.8399274488570763, 'eval_runtime': 3.2772, 'eval_samples_per_second': 30.514, 'eval_steps_per_second': 2.136, 'epoch': 3.0}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 69%|██████▉   | 100/145 [41:03<18:02, 24.06s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.7663, 'grad_norm': 7.232136249542236, 'learning_rate': 1.5517241379310346e-05, 'epoch': 3.45}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                 \n",
      " 80%|████████  | 116/145 [47:23<08:50, 18.30s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.43516409397125244, 'eval_f1': 0.8797674004703547, 'eval_runtime': 3.2975, 'eval_samples_per_second': 30.326, 'eval_steps_per_second': 2.123, 'epoch': 4.0}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                   \n",
      "100%|██████████| 145/145 [1:00:40<00:00, 19.18s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.39272159337997437, 'eval_f1': 0.8914389523348718, 'eval_runtime': 3.5564, 'eval_samples_per_second': 28.118, 'eval_steps_per_second': 1.968, 'epoch': 5.0}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 145/145 [1:00:42<00:00, 25.12s/it]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'train_runtime': 3642.7783, 'train_samples_per_second': 1.235, 'train_steps_per_second': 0.04, 'train_loss': 0.535627057634551, 'epoch': 5.0}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "events.out.tfevents.1735555878.Davids-MacBook-Pro.local.23438.0: 100%|██████████| 9.32k/9.32k [00:00<00:00, 55.0kB/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "CommitInfo(commit_url='https://huggingface.co/davidberenstein1957/domain-classifier/commit/915f4b03c230cc8f376f13729728f14347400041', commit_message='End of training', commit_description='', oid='915f4b03c230cc8f376f13729728f14347400041', pr_url=None, repo_url=RepoUrl('https://huggingface.co/davidberenstein1957/domain-classifier', endpoint='https://huggingface.co', repo_type='model', repo_id='davidberenstein1957/domain-classifier'), pr_revision=None, pr_num=None)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer.train()\n",
    "# Save processor and create model card\n",
    "tokenizer.save_pretrained(\"ModernBERT-domain-classifier\")\n",
    "trainer.create_model_card()\n",
    "trainer.push_to_hub()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We get an F1 score of 0.89 on the test set, which is pretty good for the small dataset and time spent."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run inference\n",
    "\n",
    "We can now load the model and run inference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Device set to use mps:0\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[{'label': 'health', 'score': 0.6779336333274841}]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from transformers import pipeline\n",
    " \n",
    "# load model from huggingface.co/models using our repository id\n",
    "classifier = pipeline(\n",
    "    task=\"text-classification\", \n",
    "    model=\"argilla/ModernBERT-domain-classifier\", \n",
    "    device=0,\n",
    ")\n",
    " \n",
    "sample = \"Smoking is bad for your health.\"\n",
    " \n",
    "classifier(sample)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "We have shown that we can generate a synthetic dataset from an LLM and finetune a ModernBERT model on it. This the effectiveness of synthetic data and the novel ModernBERT model, which is new and improved version of BERT models, with 8192 token context length, significantly better downstream performance, and much faster processing speeds. \n",
    "\n",
    "Pretty cool for 20 minutes of generating data, and an hour of fine-tuning on consumer hardware."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}