DeepSeek-R1-Distill-Llama-8B-Medical-COT

🏥 Fine-tuned Medical Model

This is a fine-tuned version of DeepSeek-R1-Distill-Llama-8B, optimized for medical reasoning and clinical case analysis using LoRA (Low-Rank Adaptation) with Unsloth.

• Base Model: DeepSeek-R1-Distill-Llama-8B
• Fine-Tuning Framework: Unsloth
• Dataset: FreedomIntelligence/medical-o1-reasoning-SFT
• Quantization: 4-bit (bitsandbytes)
• Task: Clinical reasoning, medical question-answering, diagnosis assistance
• Pipeline Tag: text-generation
• Metrics: loss, accuracy
• Library Name: transformers

---

📖 Model Details

Feature	Value
Architecture	Llama-8B (Distilled)
Language	English
Training Steps	60
Batch Size	2 (with gradient accumulation)
Gradient Accumulation Steps	4
Precision	Mixed (FP16/BF16 based on GPU support)
Optimizer	AdamW 8-bit
Fine-Tuned With	PEFT + LoRA (Unsloth)

---

📊 Training Summary

Loss Trend During Fine-Tuning:

Step	Training Loss
10	1.9188
20	1.4615
30	1.4023
40	1.3088
50	1.3443
60	1.3140

---

🚀 How to Use

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "develops20/DeepSeek-R1-Distill-Llama-8B-Medical-COT"

# Load model and tokenizer
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.float16, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Run inference
def ask_model(question):
    inputs = tokenizer(question, return_tensors="pt").to("cuda")
    outputs = model.generate(input_ids=inputs.input_ids, max_new_tokens=512)
    return tokenizer.decode(outputs[0], skip_special_tokens=True)

question = "A 61-year-old woman has involuntary urine loss when coughing. What would cystometry likely reveal?"
print(ask_model(question))

Example Outputs
Q: "A 59-year-old man presents with fever, night sweats, and a 12mm aortic valve vegetation. What is the most likely predisposing factor?"
Model's Answer: "The most likely predisposing factor for this patient’s infective endocarditis is a history of valvular heart disease or prosthetic valves, given the presence of an aortic valve vegetation. The causative organism is likely Enterococcus species, which does not grow in high salt concentrations."