---
license: apache-2.0
datasets:
- ehartford/dolphin
- LinhDuong/chatdoctor-200k
- sahil2801/code_instructions_120k
- medalpaca/medical_meadow_mediqa
- kaiokendev/SuperCOT-dataset
language:
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- medical
- code
---
# Model Card for Model ID

<!-- Provide a quick summary of what the model is/does. -->

This model is an instruction-tuned Open LLaMa model with 7B parameters, with specialities in medical QA and code instruction.

## Model Details

<!-- Provide a longer summary of what this model is. -->

- **Model type:** LlamaForCausalLM
- **Language(s) (NLP):** English
- **License:** Apache 2.0
- **Finetuned from model (QLoRA):** [openlm-research/open_llama_7b_v2](https://huggingface.co/openlm-research/open_llama_7b_v2)

## How to Get Started with the Model

Use the code below to get started with the model.

```py
import torch
from transformers import LlamaTokenizer, LlamaForCausalLM

model_path = 'yhyhy3/open_llama_7b_v2_med_dolphin_qlora_merged'

tokenizer = LlamaTokenizer.from_pretrained(model_path)
model = LlamaForCausalLM.from_pretrained(
    model_path, torch_dtype=torch.float16, device_map='auto',
)

prompt = '''### Instruction: Answer the following question.

### Input: What is the capital of New Jersey?

### Response:'''
input_ids = tokenizer(prompt, return_tensors="pt").input_ids

generation_output = model.generate(
    input_ids=input_ids, max_new_tokens=32
)
print(tokenizer.decode(generation_output[0]))
```

## Training Details

### Training Data

Converted the following datasets to alpaca:instruction format:
1. [ehartford/dolphin](https://huggingface.co/datasets/ehartford/dolphin)
  - ORCA style dataset generously created by [Eric Hartford](https://huggingface.co/ehartford)
  - Only used the 1 million GPT4 generated instructions file [flan1m-alpaca-uncensored.jsonl](https://huggingface.co/datasets/ehartford/dolphin/blob/main/flan1m-alpaca-uncensored.jsonl).
2. [LinhDuong/chatdoctor-200k](https://huggingface.co/datasets/LinhDuong/chatdoctor-200k)
  - Refined dataset sourced from icliniq medical QA forum
3. [sahil2801/code_instructions_120k](https://huggingface.co/datasets/sahil2801/code_instructions_120k)
  - Code instruction dataset generously created by Sahil Chaudhary from ThreeSixty AI
4. [medalpaca/medical_meadow_mediqa](https://huggingface.co/datasets/medalpaca/medical_meadow_mediqa)
  - MEDIQA is a dataset of manually generated, question-driven summaries of multi and single document answers to consumer health questions from medalpaca group.
5. [kaiokendev/SuperCOT-dataset](https://huggingface.co/datasets/kaiokendev/SuperCOT-dataset)
  - Code instruction dataset generously created by Kaio Ken

### Training Procedure 

Trained using axolotl QLoRa on RunPod 8x A6000 on Community Cloud for 2 epochs (~14 hours).

axolotl training config:
```yaml

```