File size: 4,372 Bytes
bd8523a 10f805f bd8523a 10f805f |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 |
---
license: llama2
datasets:
- nvidia/OpenMathInstruct-1
language:
- en
library_name: transformers
tags:
- nvidia
- code
- math
---
# OpenMath-CodeLlama-7b-Python
## Description:
OpenMath models were designed to solve mathematical problems by integrating text-based reasoning with code blocks
executed by Python interpreter. The models were trained on [OpenMathInstruct-1](https://huggingface.co/datasets/nvidia/OpenMathInstruct-1),
a math instruction tuning dataset with 1.8M problem-solution pairs generated using permissively licensed
[Mixtral-8x7B](https://huggingface.co/mistralai/Mixtral-8x7B-v0.1) model.
| Model | Size | GSM8K | MATH |
|--------------------------------------------------|-------|-----------|----------|
| GPT-4 [1] | - | 94.4 | 56.2 |
| GPT-4 + code [2] | - | 92.9 | 69.7 |
| OpenMath-CodeLlama-7B ([nemo](https://huggingface.co/nvidia/OpenMath-CodeLlama-7b-Python), [HF](https://huggingface.co/nvidia/OpenMath-CodeLlama-7b-Python-hf)) | 7B | 75.9 | 43.6 |
| OpenMath-CodeLlama-7B + self-consistency (k=50) | 7B | 84.8 | 55.6 |
| OpenMath-Mistral-7B ([nemo](https://huggingface.co/nvidia/OpenMath-Mistral-7B-v0.1), [HF](https://huggingface.co/nvidia/OpenMath-Mistral-7B-v0.1-hf)) | 7B | 80.2 | 44.5 |
| OpenMath-Mistral-7B + self-consistency (k=50) | 7B | 86.9 | 57.2 |
| OpenMath-CodeLlama-13B ([nemo](https://huggingface.co/nvidia/OpenMath-CodeLlama-13b-Python), [HF](https://huggingface.co/nvidia/OpenMath-CodeLlama-13b-Python-hf)) | 13B | 78.8 | 45.5 |
| OpenMath-CodeLlama-13B + self-consistency (k=50) | 13B | 86.8 | 57.6 |
| OpenMath-CodeLlama-34B ([nemo](https://huggingface.co/nvidia/OpenMath-CodeLlama-34b-Python), [HF](https://huggingface.co/nvidia/OpenMath-CodeLlama-34b-Python-hf)) | 34B | 80.7 | 48.3 |
| OpenMath-CodeLlama-34B + self-consistency (k=50) | 34B | 88.0 | 60.2 |
| OpenMath-Llama2-70B ([nemo](https://huggingface.co/nvidia/OpenMath-Llama-2-70b), [HF](https://huggingface.co/nvidia/OpenMath-Llama-2-70b-hf)) | 70B | 84.7 | 46.3 |
| OpenMath-Llama2-70B + self-consistency (k=50) | 70B | 90.1 | 58.3 |
| OpenMath-CodeLlama-70B ([nemo](https://huggingface.co/nvidia/OpenMath-CodeLlama-70b-Python), [HF](https://huggingface.co/nvidia/OpenMath-CodeLlama-70b-Python-hf)) | 70B | **84.6** | **50.7** |
| OpenMath-CodeLlama-70B + self-consistency (k=50) | 70B | **90.8** | **60.4** |
The pipeline we used to produce these models is fully open-sourced under a commercially permissive license.
- [Code](https://github.com/Kipok/NeMo-Skills)
- [Models](https://huggingface.co/collections/nvidia/openmath-65c5619de2ba059be0775014)
- [Dataset](https://huggingface.co/datasets/nvidia/OpenMathInstruct-1)
## How to use the models?
Try to [run inference with our models](/docs/inference.md) with just a few commands!
We provide [all instructions](/docs/reproducing-results.md) to fully reproduce our results.
If you want to improve your own models or to learn more about our pipeline, read through the relevant docs below.
- [Model evaluation](/docs/evaluation.md)
- [Generating synthetic data](/docs/synthetic-data-generation.md)
- [Finetuning models](/docs/finetuning.md)
## Training
This model is trained with [NVIDIA NeMo](https://www.nvidia.com/en-us/ai-data-science/generative-ai/nemo-framework/),
an end-to-end, cloud-native framework to build, customize, and deploy generative AI models anywhere.
It includes training and inferencing frameworks, guardrailing toolkits, data curation tools, and pretrained models,
offering enterprises an easy, cost-effective, and fast way to adopt generative AI.
Please see [NeMo-Skills Github repo](https://github.com/Kipok/NeMo-Skills) for training details.
## Contact
E-Mail: [Igor Gitman](mailto:[email protected])
## Citation
If you find this model useful, please cite the following works
TODO
## License
The use of this model is governed by the [Llama 2 Community License Agreement](https://ai.meta.com/llama/license/)
|