Update README.md

README.md

@@ -9,59 +9,80 @@ model-index:
   results: []
 ---
 
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
+# ModernBERT Engagement Content Regression
+### What is this?
 
-# modernBERT-content-regression
+This is an exploration of using modernBERT for the text regression task of predicting engagement metrics for text content. In this case, we are predicting the clickthrough rate (CTR) of email text content.
 
-This model is a fine-tuned version of [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) on an unknown dataset.
-It achieves the following results on the evaluation set:
-- Loss: 2.4624
-- Mse: 2.4624
-- Rmse: 1.5692
-- Mae: 1.1822
-- R2: 0.3258
-- Smape: 56.6145
+We will be exploring hyperparameter tuning of modernBert; and how to use it for regression, as well as comparing the results to a benchmark model.
 
-## Model description
+This type of task if difficult, we can remember the quote
+> “Half my advertising is wasted; the trouble is, I don't know which half”
+> -John Wanamaker
 
-More information needed
+We are also excluding other relevant factors such as the time of day the email is sent, the day of the week, the recipient, etc in this experiment.
 
-## Intended uses & limitations
+Links for project:
+- Model - [ModernBERT-Engagement-Content-Regression](https://huggingface.co/Forecast-ing/modernBERT-content-regression)
+- Training notebook - [Training Notebook](https://github.com/Forecast-ing/modernbert-content-regression/blob/main/model_training.ipynb)
+- Demo - [Demo Space](https://huggingface.co/spaces/Forecast-ing/modernbert-content-regression)
 
-More information needed
+This work is indebted to the work of many community members and blog posts.
+- [ModernBERT Announcement](https://huggingface.co/blog/modernbert)
+- [Fine-tune classifier with ModernBERT in 2025](https://www.philschmid.de/fine-tune-modern-bert-in-2025)
+- [How to set up Trainer for a regression](https://discuss.huggingface.co/t/how-to-set-up-trainer-for-a-regression/12994)
+- Additional thanks to the creators of ModernBERT!
 
-## Training and evaluation data
 
-More information needed
+### Our dataset
+We will be using a dataset of 548 emails where we have the text of the email `text` and the CTR we are trying to predict `labels`.
 
-## Training procedure
+We look forward in the improvements of ModernBERT to fine-tune models specifically for each potential users email dataset. The variability of email data, as well as the small size of the dataset pose an interesting regression challenge.
 
-### Training hyperparameters
+### Benchmarking
+We will start by using the Catboost library as a simple benchmark for text regression. For both the benchmark and the ModernBert run, we are using 'rmse' as the metric.
+We recieve the following results:
+| Metric | Value            |
+|--------|------------------|
+| MSE    | 2.552100633998035 |
+| RMSE   | 1.5975295408843102 |
+| MAE    | 1.1439370629666958 |
+| R²     | 0.30127932054387174 |
+| SMAPE  | 37.63064694052479 |
 
-The following hyperparameters were used during training:
-- learning_rate: 2.479942619764035e-05
-- train_batch_size: 4
-- eval_batch_size: 4
-- seed: 42
-- optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
-- lr_scheduler_type: linear
-- num_epochs: 5
+## Fitting the Modern Bert Model
 
-### Training results
+### Install dependencies and activate venv
+```bash
+uv sync
+source .venv/bin/activate
+```
+the following values need to be defined in the .env file
+- `HUGGINGFACE_TOKEN`
 
-| Training Loss | Epoch | Step | Validation Loss | Mse    | Rmse   | Mae    | R2      | Smape   |
-|:-------------:|:-----:|:----:|:---------------:|:------:|:------:|:------:|:-------:|:-------:|
-| 0.1152        | 1.0   | 124  | 4.0842          | 4.0842 | 2.0209 | 1.2199 | -0.1182 | 49.0235 |
-| 1.239         | 2.0   | 248  | 3.8036          | 3.8036 | 1.9503 | 1.2892 | -0.0414 | 52.7754 |
-| 27.8256       | 3.0   | 372  | 3.2460          | 3.2460 | 1.8017 | 1.1022 | 0.1113  | 51.7470 |
-| 0.0001        | 4.0   | 496  | 2.4134          | 2.4134 | 1.5535 | 1.0811 | 0.3392  | 52.2215 |
-| 0.1666        | 5.0   | 620  | 2.4624          | 2.4624 | 1.5692 | 1.1822 | 0.3258  | 56.6145 |
+### Run notebook for model fitting
 
+```bash
+uv run --with jupyter jupyter lab
+```
 
-### Framework versions
+### ModernBert Model Performance
+After running hyperparameter tuning for ModernBERT, we get the following results:
 
-- Transformers 4.48.0.dev0
-- Pytorch 2.5.1+cu124
-- Datasets 3.2.0
-- Tokenizers 0.21.0
+| Metric | Value            |
+|--------|------------------|
+| MSE    | 2.4624056816101074 |
+| RMSE   | 1.5692054300218654 |
+| MAE    | 1.182181715965271 |
+| R²     | 0.325836181640625 |
+| SMAPE  | 56.61447048187256 |
+
+We see improvements in all metrics except for SMAPE. We believe that ModernBERT would scale even better with a larger dataset; as 500 example is very low for fine-tuning and are thus happy with the performance of this evaluation.
+
+### Who are we?
+At [Forecast.ing](https://forecast.ing) we are building a platform to help users create more enriching content by automatically researching trends and generating campaign ideas with AgenticAI. 
+We generate the content, and then create fine-tuned scores of how likely we think that content will succeed.
+
+## Conclusion
+We see that ModernBERT is a powerful model for text regression. We believe that with a larger dataset, we would see even better results. We are excited to see the future of ModernBERT and how it will be used for text regression.
+If interested, I can be contacted at [email protected]