|
--- |
|
license: llama2 |
|
datasets: |
|
- nvidia/OpenMathInstruct-1 |
|
language: |
|
- en |
|
library_name: transformers |
|
tags: |
|
- nvidia |
|
- code |
|
- math |
|
--- |
|
|
|
|
|
# OpenMath-CodeLlama-7b-Python |
|
|
|
## Description: |
|
|
|
OpenMath models were designed to solve mathematical problems by integrating text-based reasoning with code blocks |
|
executed by Python interpreter. The models were trained on [OpenMathInstruct-1](https://huggingface.co/datasets/nvidia/OpenMathInstruct-1), |
|
a math instruction tuning dataset with 1.8M problem-solution pairs generated using permissively licensed |
|
[Mixtral-8x7B](https://huggingface.co/mistralai/Mixtral-8x7B-v0.1) model. |
|
|
|
|
|
| Model | Size | GSM8K | MATH | |
|
|--------------------------------------------------|-------|-----------|----------| |
|
| GPT-4 [1] | - | 94.4 | 56.2 | |
|
| GPT-4 + code [2] | - | 92.9 | 69.7 | |
|
| OpenMath-CodeLlama-7B ([nemo](https://huggingface.co/nvidia/OpenMath-CodeLlama-7b-Python), [HF](https://huggingface.co/nvidia/OpenMath-CodeLlama-7b-Python-hf)) | 7B | 75.9 | 43.6 | |
|
| OpenMath-CodeLlama-7B + self-consistency (k=50) | 7B | 84.8 | 55.6 | |
|
| OpenMath-Mistral-7B ([nemo](https://huggingface.co/nvidia/OpenMath-Mistral-7B-v0.1), [HF](https://huggingface.co/nvidia/OpenMath-Mistral-7B-v0.1-hf)) | 7B | 80.2 | 44.5 | |
|
| OpenMath-Mistral-7B + self-consistency (k=50) | 7B | 86.9 | 57.2 | |
|
| OpenMath-CodeLlama-13B ([nemo](https://huggingface.co/nvidia/OpenMath-CodeLlama-13b-Python), [HF](https://huggingface.co/nvidia/OpenMath-CodeLlama-13b-Python-hf)) | 13B | 78.8 | 45.5 | |
|
| OpenMath-CodeLlama-13B + self-consistency (k=50) | 13B | 86.8 | 57.6 | |
|
| OpenMath-CodeLlama-34B ([nemo](https://huggingface.co/nvidia/OpenMath-CodeLlama-34b-Python), [HF](https://huggingface.co/nvidia/OpenMath-CodeLlama-34b-Python-hf)) | 34B | 80.7 | 48.3 | |
|
| OpenMath-CodeLlama-34B + self-consistency (k=50) | 34B | 88.0 | 60.2 | |
|
| OpenMath-Llama2-70B ([nemo](https://huggingface.co/nvidia/OpenMath-Llama-2-70b), [HF](https://huggingface.co/nvidia/OpenMath-Llama-2-70b-hf)) | 70B | 84.7 | 46.3 | |
|
| OpenMath-Llama2-70B + self-consistency (k=50) | 70B | 90.1 | 58.3 | |
|
| OpenMath-CodeLlama-70B ([nemo](https://huggingface.co/nvidia/OpenMath-CodeLlama-70b-Python), [HF](https://huggingface.co/nvidia/OpenMath-CodeLlama-70b-Python-hf)) | 70B | **84.6** | **50.7** | |
|
| OpenMath-CodeLlama-70B + self-consistency (k=50) | 70B | **90.8** | **60.4** | |
|
|
|
|
|
The pipeline we used to produce these models is fully open-sourced under a commercially permissive license. |
|
|
|
- [Code](https://github.com/Kipok/NeMo-Skills) |
|
- [Models](https://huggingface.co/collections/nvidia/openmath-65c5619de2ba059be0775014) |
|
- [Dataset](https://huggingface.co/datasets/nvidia/OpenMathInstruct-1) |
|
|
|
## How to use the models? |
|
|
|
Try to [run inference with our models](/docs/inference.md) with just a few commands! |
|
|
|
We provide [all instructions](/docs/reproducing-results.md) to fully reproduce our results. |
|
|
|
If you want to improve your own models or to learn more about our pipeline, read through the relevant docs below. |
|
|
|
- [Model evaluation](/docs/evaluation.md) |
|
- [Generating synthetic data](/docs/synthetic-data-generation.md) |
|
- [Finetuning models](/docs/finetuning.md) |
|
|
|
## Training |
|
|
|
This model is trained with [NVIDIA NeMo](https://www.nvidia.com/en-us/ai-data-science/generative-ai/nemo-framework/), |
|
an end-to-end, cloud-native framework to build, customize, and deploy generative AI models anywhere. |
|
It includes training and inferencing frameworks, guardrailing toolkits, data curation tools, and pretrained models, |
|
offering enterprises an easy, cost-effective, and fast way to adopt generative AI. |
|
|
|
Please see [NeMo-Skills Github repo](https://github.com/Kipok/NeMo-Skills) for training details. |
|
|
|
## Contact |
|
|
|
E-Mail: [Igor Gitman](mailto:[email protected]) |
|
|
|
## Citation |
|
|
|
If you find this model useful, please cite the following works |
|
|
|
TODO |
|
|
|
## License |
|
The use of this model is governed by the [Llama 2 Community License Agreement](https://ai.meta.com/llama/license/) |
|
|
