README.md

---
license: apache-2.0
datasets:
- TIGER-Lab/MATH-plus
language:
- en
tags:
- torchtune
- minerva-math
library_name: transformers
pipeline_tag: text-generation
---

# jrc/phi3-mini-math

<!-- Provide a quick summary of what the model is/does. -->

Math majors - who needs em? This model can answer any math questions you have.

## How to Get Started with the Model

Use the code below to get started with the model.

```python
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("jrc/phi3-mini-math", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("jrc/phi3-mini-math", trust_remote_code=True)
```

## Training Details

Phi3 was trained using [torchtune](https://github.com/pytorch/torchtune) and the training script + config file are located in this repository.

```bash
tune run lora_finetune_distributed.py --config mini_lora.yaml 
```

### Training Data

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

This model was finetuned on the following datasets:

* TIGER-Lab/MATH-plus: An advanced math-specific dataset with 894k samples.


#### Hardware

4 x NVIDIA A100 GPUs

Max VRAM used per GPU: 29 GB
Real time: 12 hours

## Evaluation

The finetuned model is evaluated on [minerva-math](https://research.google/blog/minerva-solving-quantitative-reasoning-problems-with-language-models/) using [EleutherAI Eval Harness](https://github.com/EleutherAI/lm-evaluation-harness) through torchtune.

```bash
tune run eleuther_eval --config eleuther_evaluation \
          checkpoint.checkpoint_dir=./lora-phi3-math \
          tasks=["minerva_math"] \
          batch_size=32 
```


|               Tasks                |Version|Filter|n-shot|  Metric   |Value |   |Stderr|
|------------------------------------|-------|------|-----:|-----------|-----:|---|-----:|
|minerva_math                        |N/A    |none  |     4|exact_match|0.1670|±  |0.0051|
| - minerva_math_algebra             |      1|none  |     4|exact_match|0.2502|±  |0.0126|
| - minerva_math_counting_and_prob   |      1|none  |     4|exact_match|0.1329|±  |0.0156|
| - minerva_math_geometry            |      1|none  |     4|exact_match|0.1232|±  |0.0150|
| - minerva_math_intermediate_algebra|      1|none  |     4|exact_match|0.0576|±  |0.0078|
| - minerva_math_num_theory          |      1|none  |     4|exact_match|0.1148|±  |0.0137|
| - minerva_math_prealgebra          |      1|none  |     4|exact_match|0.3077|±  |0.0156|
| - minerva_math_precalc             |      1|none  |     4|exact_match|0.0623|±  |0.0104|


## Model Card Contact

[More Information Needed]