File size: 3,585 Bytes
c357eae e158318 78d632a 3a09f86 c357eae |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 |
---
base_model:
- deepseek-ai/deepseek-coder-1.3b-base
finetuned version: aiswaryards/deepseek1.3B-coder-dora-finetuned
tags:
- '`DoRA`'
- '`question-generation`'
- '`data-science`'
- '`knowledge-transfer`'
- '`rag`'
- '`llm`'
- '`agents`'
- '`deepseek`'
model_type: causal-lm
---
## Quick Overview:
The deployed model `aiswaryards/deepseek1.3B-coder-dora-finetuned` is a fine-tuned version of [DeepSeek-Coder 1.3B](https://huggingface.co/deepseek-ai/deepseek-coder-1.3b-base), adapted using the **DoRA (Decoupled Low-Rank Adaptation)** method.
It was trained as part of a **graduate-level academic project** focused on simulating expert question generation from domain-specific Knowledge Transfer (KT) documents.
To fine-tune a compact and capable LLM to generate **high-quality, context-specific technical questions** based on internal handover documents in a Retrieval-Augmented Generation (RAG) pipeline.
## Use Case
- Designed to simulate domain expert handover, where the model generates precise and context-aware questions to extract undocumented insights from KT documents.
- Create prompts that makes the
## Model Information:
Base: deepseek-ai/deepseek-coder-1.3b-base
Fine-tuning: (DoRA) Dynamically Optimized Rank Adaptation
Format: Instruction → Input (context) → Response (question)
## Fine-Tuning Details
- **Base Model**: `deepseek-ai/deepseek-coder-1.3b-base`
- **Adapter Method**: DoRA using `AdaLoraConfig` from PEFT
- `r`: 8
- `lora_alpha`: 16
- `lora_dropout`: 0.05
- `bias`: none
- `target_modules`: ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]
- use_dora=True
- **Quantization**: bfloat16
- **Data Source**: Webscrapped dataset (~100+ high-quality Q&A pairs from Knowledge Transfer docs)
- **Training**:
- Epochs: 5
- Batch Size: 1 (gradient accumulation: 8)
- Optimized using Hugging Face `Trainer`
- Platform: Google Colab (A100)
- Quantization: 16-bit (bfloat16)
- Hardware: Google Colab - (A100 GPU) - cuda
- Frameworks: HuggingFace Transformers, PEFT, Datasets, Trainer
- Tokenization length: 1024
- Trained on : high-quality Q&A pairs collected from various branches of data science domain KT handovers.
## Training Performance:
Step | Train Loss | Validation Loss
=======================================
400 | 1.780 | 1.795
=======================================
- This exhibits a strong convergence for a fine-tuned instruction.
- The validation loss curve flattens under 2.0, which is a great sign for generative quality.
## Use Cases
- Domain Expert Simulation (Agentic RAG)
- Knowledge Transfer Automation
- Multiple roles - currently experimented on datascience domain (for eg. BI / Data Engineering role transitions)
- Question synthesis for downstream QA chains
## Challenges:
- It requires instruction-style prompting.
- Might hallucinate if context is vague or unrelated
- It is best suited within structured RAG pipelines
## License
---
license: deepseek
---
### DeepSeek-Coder 1.3B - DoRA Fine-tuned Version
This model is a fine-tuned version of `deepseek-ai/deepseek-coder-1.3b-base` using the DoRA method on academic Q&A data from Knowledge Transfer documents.
...
This model inherits the original license from [DeepSeek-AI](https://huggingface.co/deepseek-ai/deepseek-coder-1.3b-base), which can be found [here](https://huggingface.co/deepseek-ai/deepseek-coder-1.3b-base/blob/main/LICENSE).
For academic use only. Please refer to the original license terms for reuse, distribution, or commercial applications.
|