Update README.md to include detailed model information for FinSight AI, a financial advisory chatbot. Added sections on model details, usage examples, training details, limitations, and future improvements. Changed license from Apache-2.0 to MIT and updated language and tags for better categorization.

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Zahemen
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -1,3 +1,234 @@
----
-license: apache-2.0
----
+---
+language:
+  - en
+license: mit
+library_name: transformers
+tags:
+  - financial-analysis
+  - conversational
+  - finance
+  - qlora
+  - financial-advice
+  - text-generation
+pipeline_tag: text-generation
+model-index:
+  - name: FinSight AI
+    results:
+      - task:
+          type: text-generation
+          name: Financial Advisory Generation
+        dataset:
+          type: custom
+          name: Financial Conversations
+        metrics:
+          - type: rouge1
+            value: 12.57%
+            name: ROUGE-1 Improvement
+          - type: rouge2
+            value: 79.48%
+            name: ROUGE-2 Improvement
+          - type: rougeL
+            value: 24.00%
+            name: ROUGE-L Improvement
+          - type: bleu
+            value: 135.36%
+            name: BLEU Improvement
+base_model: HuggingFaceTB/SmolLM2-1.7B-Instruct
+---
+
+# FinSight AI - Financial Advisory Chatbot
+
+A fine-tuned version of SmolLM2-1.7B optimized for financial advice and discussion.
+
+<div align="center">
+  <a href="https://pytorch.org/" target="_blank">
+    <img src="https://img.shields.io/badge/PyTorch-EE4C2C?style=for-the-badge&logo=pytorch&logoColor=white" alt="PyTorch" />
+  </a>
+  <a href="https://huggingface.co/transformers/" target="_blank">
+    <img src="https://img.shields.io/badge/🤗%20Transformers-FFAE33?style=for-the-badge&logoColor=white" alt="Transformers" />
+  </a>
+  <a href="https://huggingface.co/" target="_blank">
+    <img src="https://img.shields.io/badge/🤗%20Hugging%20Face-0050C5?style=for-the-badge&logoColor=white" alt="Hugging Face" />
+  </a>
+  <a href="https://github.com/microsoft/LoRA" target="_blank">
+    <img src="https://img.shields.io/badge/LoRA-2088FF?style=for-the-badge&logo=github&logoColor=white" alt="LoRA" />
+  </a>
+  <a href="https://github.com/TimDettmers/bitsandbytes" target="_blank">
+    <img src="https://img.shields.io/badge/BitsAndBytes-4D4D4D?style=for-the-badge&logo=github&logoColor=white" alt="BitsAndBytes" />
+  </a>
+</div>
+
+
+<div align="center">
+  <h4><a href="https://github.com/zahemen9900/Datasets-for-Finsight/blob/97d7cacfff62e7b6099ef3bb0af9cf3d044a5b35/metrics/model_paper.md" target="_blank">📄 Read Model Paper 📄</a></h4>
+</div>
+
+## Model Details
+
+- **Base Model**: HuggingFaceTB/SmolLM2-1.7B-Instruct
+- **Task**: Financial Advisory and Discussion
+- **Training Data**: Curated dataset of ~11,000 financial conversations (~16.5M tokens)
+- **Training Method**: QLoRA (4-bit quantization with LoRA)
+- **Language**: English
+- **License**: MIT
+
+## Model Description
+
+FinSight AI is a specialized financial advisory assistant built by fine-tuning SmolLM2-1.7B-Instruct using QLoRA (Quantized Low-Rank Adaptation). The model has been trained on a comprehensive dataset of financial conversations to provide accurate, concise, and helpful information across various financial domains including personal finance, investing, market analysis, and financial planning.
+
+Our evaluation demonstrates significant performance improvements across all standard NLP metrics **(ROUGE-1 , ROUGE-2, ROUGE-L & BLEU)**, showcasing the effectiveness of our domain-specific training approach. The model exhibits enhanced capabilities with richer financial terminology usage, more precise responses, improved handling of numerical data, and greater technical accuracy - all while maintaining a compact, resource-efficient architecture suitable for deployment on consumer hardware.
+
+## Usage
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
+import torch
+from peft import PeftModel
+
+# For 4-bit quantized inference (recommended)
+bnb_config = BitsAndBytesConfig(
+  load_in_4bit=True,
+  bnb_4bit_use_double_quant=True,
+  bnb_4bit_quant_type="nf4",
+  bnb_4bit_compute_dtype=torch.bfloat16
+)
+
+# First load the base model with quantization
+base_model = AutoModelForCausalLM.from_pretrained(
+  "HuggingFaceTB/SmolLM2-1.7B-Instruct",
+  quantization_config=bnb_config,
+  device_map="auto"
+)
+
+# Then load the adapter weights (LoRA)
+model = PeftModel.from_pretrained(base_model, "zahemen9900/finsight-ai")
+tokenizer = AutoTokenizer.from_pretrained("HuggingFaceTB/SmolLM2-1.7B-Instruct")
+
+# Example usage
+prompt = "What's a good strategy for long-term investing?"
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+outputs = model.generate(
+  inputs.input_ids,
+  max_new_tokens=512,
+  temperature=0.7,
+  top_p=0.95,
+  do_sample=True
+)
+response = tokenizer.decode(outputs[0], skip_special_tokens=True)
+print(response)
+```
+
+### For streaming options: 
+
+```python
+from transformers import TextIteratorStreamer
+import threading
+
+# Setup model and tokenizer (same as above)
+
+prompt = "What's a good strategy for long-term investing?"
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+
+# Create a streamer
+streamer = TextIteratorStreamer(tokenizer, timeout=10.0, skip_prompt=True, skip_special_tokens=True)
+
+# Generate in a separate thread
+generation_kwargs = {
+    "input_ids": inputs.input_ids,
+    "max_new_tokens": 512,
+    "temperature": 0.7,
+    "top_p": 0.95,
+    "do_sample": True,
+    "streamer": streamer
+}
+
+thread = threading.Thread(target=model.generate, kwargs=generation_kwargs)
+thread.start()
+
+# Iterate over the generated text
+print("Response: ", end="")
+for text in streamer:
+    print(text, end="", flush=True)
+
+```
+
+## Example Interactions
+
+**Query**: "What is dollar-cost averaging?"
+
+**Response**: "Dollar-cost averaging (DCA) is an investment strategy in which an investor divides the total amount to be invested into equal parts and invests those parts at regular intervals, regardless of the asset's price. This approach aims to reduce the impact of market volatility by spreading out the investment over time. By purchasing more shares when prices are low and fewer shares when prices are high, DCA can potentially lower the average cost per share over the long term."
+
+**Query**: "How do I start investing with a small budget?"
+
+**Response**: "Starting to invest with a small budget is definitely possible. Here are some steps to get you started:
+
+1. **Set clear goals**: Determine what you're investing for and your time horizon.
+
+2. **Build an emergency fund first**: Before investing, ensure you have 3-6 months of living expenses saved.
+
+3. **Use micro-investing apps**: Platforms like Acorns, Stash, or Robinhood allow you to start with very small amounts.
+
+4. **Consider index ETFs**: These offer instant diversification and typically have low expense ratios.
+
+5. **Utilize employer retirement plans**: If available, contribute to a 401(k), especially if your employer offers matching contributions.
+
+6. **Implement dollar-cost averaging**: Invest fixed amounts regularly rather than trying to time the market.
+
+7. **Reinvest dividends**: This allows your investments to compound over time.
+
+8. **Be consistent**: Small regular investments can grow significantly over time due to compound interest.
+
+Remember, the most important factor is getting started early and staying consistent, rather than the initial amount you invest."
+
+## Training Details
+
+The model was trained using the following configuration:
+
+- **QLoRA Parameters**: 
+  - Rank (r): 64
+  - Alpha: 16
+  - Target modules: Query, Key, Value projections, MLP layers
+  - 4-bit NF4 quantization with double quantization
+
+- **Training Hyperparameters**:
+  - Learning rate: 2e-4
+  - Epochs: 2
+  - Batch size: 2 (with gradient accumulation steps of 4)
+  - Weight decay: 0.05
+  - Scheduler: Cosine with restarts
+  - Warmup ratio: 0.15
+
+- **Hardware**: Consumer-grade NVIDIA RTX 3050 GPU with 6GB VRAM
+
+## Limitations
+
+## Limitations
+
+- **Information Currency**: Financial data and knowledge within the model is limited to the training data cutoff date. Market conditions, regulations, and financial instruments may have changed since then.
+- **No Real-time Information**: The model operates without internet connectivity and cannot access current market data, breaking news, or recent economic developments.
+- **Not Financial Advice**: Responses should not be considered personalized financial advice. The model cannot account for individual financial situations, risk tolerances, or specific circumstances required for proper financial planning.
+- **Language Limitations**: While optimized for English financial terminology, the model may have reduced performance with non-English financial terms or concepts specific to regional markets.
+- **Regulatory Compliance**: The model is not updated with the latest financial regulations across different jurisdictions and cannot ensure compliance with local financial laws.
+- **Complexity Handling**: May struggle with highly complex or niche financial scenarios that were underrepresented in the training data.
+
+## Future Improvements
+
+- **Retrieval Augmented Generation (RAG)**: Implementing RAG would allow the model to reference current financial data, market statistics, and regulatory information before generating responses, significantly improving accuracy and relevance.
+- **Domain-Specific Fine-tuning**: Additional training on specialized financial domains like cryptocurrency, derivatives trading, and international tax regulations.
+- **Multilingual Support**: Expanding capabilities to handle financial terminology and concepts across multiple languages and markets.
+- **Personalization Framework**: Developing mechanisms to better contextualize responses based on stated user preferences while maintaining privacy.
+
+## Citation
+
+If you use FinSight AI in your research, please cite:
+```md
+
+@misc{FinSightAI2025,
+  author = {Zahemen, FinsightAI Team},
+  title = {FinSight AI: Enhancing Financial Domain Performance of Small Language Models Through QLoRA Fine-tuning},
+  year = {2025},
+  publisher = {GitHub},
+  journal = {GitHub repository},
+  howpublished = {\url{https://github.com/zahemen9900/FinsightAI}}
+}
+```

adapter_config.json

@@ -0,0 +1,31 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "HuggingFaceTB/SmolLM2-1.7B-Instruct",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": "gaussian",
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "lora_alpha": 16,
+  "lora_dropout": 0.05,
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "v_proj",
+    "mixer_mlp",
+    "gate_proj",
+    "mixer_self_attention",
+    "up_proj",
+    "k_proj",
+    "mixer_cross_attention",
+    "q_proj",
+    "o_proj",
+    "down_proj"
+  ],
+  "task_type": "CAUSAL_LM"
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3fa488100b3b6a79d43bc6742f1938de7d2d14ad50dfc6b831faa2beb3056240
+size 289452128

all_results.json

@@ -0,0 +1,12 @@
+{
+    "epoch": 1.4679393049437102,
+    "eval_loss": 1.5792005062103271,
+    "eval_runtime": 1884.1302,
+    "eval_samples_per_second": 1.446,
+    "eval_steps_per_second": 0.723,
+    "total_flos": 1.105565365842985e+17,
+    "train_loss": 1.763797264099121,
+    "train_runtime": 30390.5778,
+    "train_samples_per_second": 0.395,
+    "train_steps_per_second": 0.049
+}

eval_results.json

@@ -0,0 +1,7 @@
+{
+    "epoch": 1.4679393049437102,
+    "eval_loss": 1.5792005062103271,
+    "eval_runtime": 1884.1302,
+    "eval_samples_per_second": 1.446,
+    "eval_steps_per_second": 0.723
+}

inference.py

@@ -0,0 +1,212 @@
+#!/usr/bin/env python3
+"""
+FinSight AI - Inference script for financial advisory chatbot
+This script provides a simple way to interact with the model via command line
+"""
+
+import os
+import argparse
+import torch
+from typing import List, Dict
+from transformers import (
+    AutoModelForCausalLM, 
+    AutoTokenizer, 
+    TextStreamer,
+    BitsAndBytesConfig
+)
+
+class FinancialAdvisor:
+    def __init__(
+        self,
+        model_id: str = "zahemen9900/finsight-ai",
+        use_4bit: bool = True,
+        device: str = None
+    ):
+        self.device = device or ("cuda" if torch.cuda.is_available() else "cpu")
+        print(f"Using device: {self.device}")
+        
+        # Configure quantization if requested and available
+        if use_4bit and self.device == "cuda":
+            print("Loading model in 4-bit quantization mode")
+            bnb_config = BitsAndBytesConfig(
+                load_in_4bit=True,
+                bnb_4bit_quant_type="nf4",
+                bnb_4bit_use_double_quant=True,
+                bnb_4bit_compute_dtype=torch.bfloat16
+            )
+        else:
+            print("Loading model in standard mode")
+            bnb_config = None
+        
+        # Load tokenizer and model
+        self.tokenizer = AutoTokenizer.from_pretrained(model_id)
+        self.model = AutoModelForCausalLM.from_pretrained(
+            model_id,
+            quantization_config=bnb_config,
+            device_map="auto" if self.device == "cuda" else None,
+            torch_dtype=torch.bfloat16 if self.device == "cuda" else torch.float32,
+        )
+        
+        if self.device == "cpu":
+            self.model = self.model.to(self.device)
+            
+        self.model.eval()
+        self.conversation_history = []
+        self.system_message = {
+            "role": "system",
+            "content": (
+                "You are FinSight AI, a helpful and knowledgeable financial assistant. "
+                "You can provide information and guidance on financial topics, market trends, investment strategies, "
+                "and personal finance management. Always strive to be accurate, informative, and helpful. "
+                "Remember that you cannot provide personalized financial advice that would require knowing a person's "
+                "complete financial situation or future market movements."
+            )
+        }
+    
+    def generate_response(
+        self,
+        prompt: str,
+        temperature: float = 0.7,
+        max_new_tokens: int = 512,
+        stream: bool = True
+    ) -> str:
+        """Generate response from the model"""
+        # Manage conversation history (keep last 5 exchanges)
+        if len(self.conversation_history) > 10:
+            self.conversation_history = self.conversation_history[-10:]
+            
+        # Create messages with history
+        messages = [self.system_message] + self.conversation_history
+        messages.append({"role": "user", "content": prompt})
+        
+        # Format prompt using chat template
+        formatted_prompt = self.tokenizer.apply_chat_template(
+            messages, 
+            tokenize=False,
+            add_generation_prompt=True
+        )
+        
+        # Encode the input
+        inputs = self.tokenizer(
+            formatted_prompt, 
+            return_tensors="pt",
+            truncation=True,
+            max_length=4096
+        ).to(self.device)
+        
+        # Setup streamer if requested
+        streamer = TextStreamer(
+            self.tokenizer, 
+            skip_prompt=True,
+            skip_special_tokens=True
+        ) if stream else None
+        
+        # Generate response
+        with torch.inference_mode():
+            output_ids = self.model.generate(
+                inputs.input_ids,
+                attention_mask=inputs.attention_mask,
+                max_new_tokens=max_new_tokens,
+                do_sample=True,
+                temperature=temperature,
+                top_p=0.95,
+                streamer=streamer,
+                pad_token_id=self.tokenizer.eos_token_id,
+                repetition_penalty=1.1
+            )
+        
+        # Return the response
+        if not stream:
+            response = self.tokenizer.decode(
+                output_ids[0][inputs.input_ids.shape[1]:], 
+                skip_special_tokens=True
+            )
+            print("\nAssistant:", response)
+        else:
+            response = ""  # Response was already streamed
+            
+        # Update conversation history
+        self.conversation_history.append({"role": "user", "content": prompt})
+        self.conversation_history.append({"role": "assistant", "content": response if response else "[Response was streamed]"})
+        
+        return response
+        
+    def start_chat_loop(self):
+        """Start an interactive chat session"""
+        print("\nWelcome to FinSight AI - Your Financial Advisory Assistant!")
+        print("Type 'quit', 'exit', or press Ctrl+C to end the conversation.\n")
+        
+        while True:
+            try:
+                user_input = input("\nYou: ").strip()
+                if user_input.lower() in ["quit", "exit", "q"]:
+                    break
+                    
+                if user_input.lower() == "clear":
+                    self.conversation_history = []
+                    print("Conversation history cleared.")
+                    continue
+                
+                print("\nAssistant: ", end="", flush=True)
+                self.generate_response(user_input)
+                
+            except KeyboardInterrupt:
+                print("\nExiting chat...")
+                break
+            except Exception as e:
+                print(f"\nError: {e}")
+                continue
+                
+        print("\nThank you for using FinSight AI. Goodbye!")
+
+def main():
+    parser = argparse.ArgumentParser(description="FinSight AI Inference Script")
+    parser.add_argument(
+        "--model_id", 
+        type=str, 
+        default="zahemen9900/finsight-ai",
+        help="Model ID or path to load"
+    )
+    parser.add_argument(
+        "--no_quantize", 
+        action="store_true",
+        help="Disable 4-bit quantization (uses more memory)"
+    )
+    parser.add_argument(
+        "--query", 
+        type=str,
+        help="Single query mode: provide a question and get one response"
+    )
+    parser.add_argument(
+        "--temperature", 
+        type=float, 
+        default=0.7,
+        help="Sampling temperature (higher = more random)"
+    )
+    parser.add_argument(
+        "--max_tokens", 
+        type=int, 
+        default=512,
+        help="Maximum number of new tokens to generate"
+    )
+    
+    args = parser.parse_args()
+    
+    advisor = FinancialAdvisor(
+        model_id=args.model_id, 
+        use_4bit=not args.no_quantize
+    )
+    
+    # Single query mode
+    if args.query:
+        advisor.generate_response(
+            args.query, 
+            temperature=args.temperature, 
+            max_new_tokens=args.max_tokens
+        )
+    # Interactive chat mode
+    else:
+        advisor.start_chat_loop()
+
+if __name__ == "__main__":
+    main()

requirements.txt

@@ -0,0 +1,11 @@
+torch>=2.0.0
+transformers>=4.34.0
+accelerate>=0.30.0
+peft>=0.6.2
+bitsandbytes>=0.40.0
+sentencepiece
+protobuf
+einops
+regex
+numpy
+tqdm

special_tokens_map.json

@@ -0,0 +1,34 @@
+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>"
+  ],
+  "bos_token": {
+    "content": "<|im_start|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,155 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<repo_name>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "<reponame>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "5": {
+      "content": "<file_sep>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "6": {
+      "content": "<filename>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "7": {
+      "content": "<gh_stars>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "8": {
+      "content": "<issue_start>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "9": {
+      "content": "<issue_comment>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "10": {
+      "content": "<issue_closed>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "11": {
+      "content": "<jupyter_start>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "12": {
+      "content": "<jupyter_text>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "13": {
+      "content": "<jupyter_code>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "14": {
+      "content": "<jupyter_output>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "15": {
+      "content": "<jupyter_script>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "16": {
+      "content": "<empty_output>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>"
+  ],
+  "bos_token": "<|im_start|>",
+  "chat_template": "{% for message in messages %}{% if loop.first and messages[0]['role'] != 'system' %}{{ '<|im_start|>system\nYou are a helpful AI assistant named SmolLM, trained by Hugging Face<|im_end|>\n' }}{% endif %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "extra_special_tokens": {},
+  "model_max_length": 8192,
+  "pad_token": "<|im_end|>",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>",
+  "vocab_size": 49152
+}

train_results.json

@@ -0,0 +1,8 @@
+{
+    "epoch": 1.4679393049437102,
+    "total_flos": 1.105565365842985e+17,
+    "train_loss": 1.763797264099121,
+    "train_runtime": 30390.5778,
+    "train_samples_per_second": 0.395,
+    "train_steps_per_second": 0.049
+}

trainer_state.json

@@ -0,0 +1,326 @@
+{
+  "best_metric": 1.5792005062103271,
+  "best_model_checkpoint": "qlora_output/checkpoint-1200",
+  "epoch": 1.4679393049437102,
+  "eval_steps": 600,
+  "global_step": 1500,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.03915810083210964,
+      "grad_norm": 0.07319402694702148,
+      "learning_rate": 3.555555555555556e-05,
+      "loss": 2.4428,
+      "step": 40
+    },
+    {
+      "epoch": 0.07831620166421928,
+      "grad_norm": 0.04330237954854965,
+      "learning_rate": 7.111111111111112e-05,
+      "loss": 2.268,
+      "step": 80
+    },
+    {
+      "epoch": 0.11747430249632893,
+      "grad_norm": 0.05867455527186394,
+      "learning_rate": 0.00010666666666666667,
+      "loss": 2.1806,
+      "step": 120
+    },
+    {
+      "epoch": 0.15663240332843856,
+      "grad_norm": 0.06936266273260117,
+      "learning_rate": 0.00014222222222222224,
+      "loss": 2.0778,
+      "step": 160
+    },
+    {
+      "epoch": 0.19579050416054822,
+      "grad_norm": 0.08056484907865524,
+      "learning_rate": 0.00017777777777777779,
+      "loss": 2.0382,
+      "step": 200
+    },
+    {
+      "epoch": 0.23494860499265785,
+      "grad_norm": 0.0779654011130333,
+      "learning_rate": 0.0001999317060143023,
+      "loss": 1.9227,
+      "step": 240
+    },
+    {
+      "epoch": 0.2741067058247675,
+      "grad_norm": 0.11802724003791809,
+      "learning_rate": 0.00019908312530915603,
+      "loss": 1.9139,
+      "step": 280
+    },
+    {
+      "epoch": 0.3132648066568771,
+      "grad_norm": 0.0852489247918129,
+      "learning_rate": 0.00019727282722446047,
+      "loss": 1.9423,
+      "step": 320
+    },
+    {
+      "epoch": 0.3524229074889868,
+      "grad_norm": 0.1409972459077835,
+      "learning_rate": 0.00019451838281608197,
+      "loss": 1.8484,
+      "step": 360
+    },
+    {
+      "epoch": 0.39158100832109644,
+      "grad_norm": 0.11129080504179001,
+      "learning_rate": 0.00019084652718195238,
+      "loss": 1.7694,
+      "step": 400
+    },
+    {
+      "epoch": 0.43073910915320607,
+      "grad_norm": 0.10179898887872696,
+      "learning_rate": 0.00018629289996673897,
+      "loss": 1.8026,
+      "step": 440
+    },
+    {
+      "epoch": 0.4698972099853157,
+      "grad_norm": 0.14124783873558044,
+      "learning_rate": 0.00018090169943749476,
+      "loss": 1.8217,
+      "step": 480
+    },
+    {
+      "epoch": 0.5090553108174254,
+      "grad_norm": 0.16184218227863312,
+      "learning_rate": 0.0001747252534878891,
+      "loss": 1.7847,
+      "step": 520
+    },
+    {
+      "epoch": 0.548213411649535,
+      "grad_norm": 0.11349498480558395,
+      "learning_rate": 0.00016782351173492342,
+      "loss": 1.6622,
+      "step": 560
+    },
+    {
+      "epoch": 0.5873715124816447,
+      "grad_norm": 0.08884529024362564,
+      "learning_rate": 0.00016026346363792567,
+      "loss": 1.7633,
+      "step": 600
+    },
+    {
+      "epoch": 0.5873715124816447,
+      "eval_loss": 1.6572695970535278,
+      "eval_runtime": 1912.2507,
+      "eval_samples_per_second": 1.425,
+      "eval_steps_per_second": 0.713,
+      "step": 600
+    },
+    {
+      "epoch": 0.6265296133137542,
+      "grad_norm": 0.09996389597654343,
+      "learning_rate": 0.0001521184882876585,
+      "loss": 1.6764,
+      "step": 640
+    },
+    {
+      "epoch": 0.6656877141458639,
+      "grad_norm": 0.12769252061843872,
+      "learning_rate": 0.00014346764217659653,
+      "loss": 1.7871,
+      "step": 680
+    },
+    {
+      "epoch": 0.7048458149779736,
+      "grad_norm": 0.13380451500415802,
+      "learning_rate": 0.00013439489186339282,
+      "loss": 1.7167,
+      "step": 720
+    },
+    {
+      "epoch": 0.7440039158100832,
+      "grad_norm": 0.11822285503149033,
+      "learning_rate": 0.0001249882989794231,
+      "loss": 1.6789,
+      "step": 760
+    },
+    {
+      "epoch": 0.7831620166421929,
+      "grad_norm": 0.12109290808439255,
+      "learning_rate": 0.00011533916548786857,
+      "loss": 1.583,
+      "step": 800
+    },
+    {
+      "epoch": 0.8223201174743024,
+      "grad_norm": 0.12838001549243927,
+      "learning_rate": 0.000105541147491597,
+      "loss": 1.7412,
+      "step": 840
+    },
+    {
+      "epoch": 0.8614782183064121,
+      "grad_norm": 0.16042716801166534,
+      "learning_rate": 9.568934619137046e-05,
+      "loss": 1.6519,
+      "step": 880
+    },
+    {
+      "epoch": 0.9006363191385218,
+      "grad_norm": 0.1427149474620819,
+      "learning_rate": 8.587938481769089e-05,
+      "loss": 1.6598,
+      "step": 920
+    },
+    {
+      "epoch": 0.9397944199706314,
+      "grad_norm": 0.118178591132164,
+      "learning_rate": 7.620648049573815e-05,
+      "loss": 1.7378,
+      "step": 960
+    },
+    {
+      "epoch": 0.9789525208027411,
+      "grad_norm": 0.1253277212381363,
+      "learning_rate": 6.676452005203406e-05,
+      "loss": 1.6451,
+      "step": 1000
+    },
+    {
+      "epoch": 1.0176211453744493,
+      "grad_norm": 0.15462452173233032,
+      "learning_rate": 5.764514873320761e-05,
+      "loss": 1.6475,
+      "step": 1040
+    },
+    {
+      "epoch": 1.056779246206559,
+      "grad_norm": 0.106235072016716,
+      "learning_rate": 4.893688068190932e-05,
+      "loss": 1.6686,
+      "step": 1080
+    },
+    {
+      "epoch": 1.0959373470386686,
+      "grad_norm": 0.09717393666505814,
+      "learning_rate": 4.072423980374452e-05,
+      "loss": 1.6824,
+      "step": 1120
+    },
+    {
+      "epoch": 1.1350954478707782,
+      "grad_norm": 0.13711334764957428,
+      "learning_rate": 3.308693936411421e-05,
+      "loss": 1.6147,
+      "step": 1160
+    },
+    {
+      "epoch": 1.174253548702888,
+      "grad_norm": 0.1265803724527359,
+      "learning_rate": 2.6099108277934103e-05,
+      "loss": 1.6174,
+      "step": 1200
+    },
+    {
+      "epoch": 1.174253548702888,
+      "eval_loss": 1.5792005062103271,
+      "eval_runtime": 1903.0333,
+      "eval_samples_per_second": 1.432,
+      "eval_steps_per_second": 0.716,
+      "step": 1200
+    },
+    {
+      "epoch": 1.2134116495349976,
+      "grad_norm": 0.09578167647123337,
+      "learning_rate": 1.982857160199334e-05,
+      "loss": 1.6246,
+      "step": 1240
+    },
+    {
+      "epoch": 1.2525697503671072,
+      "grad_norm": 0.14227357506752014,
+      "learning_rate": 1.4336192213613742e-05,
+      "loss": 1.5548,
+      "step": 1280
+    },
+    {
+      "epoch": 1.2917278511992167,
+      "grad_norm": 0.1526080220937729,
+      "learning_rate": 9.675280065387116e-06,
+      "loss": 1.5454,
+      "step": 1320
+    },
+    {
+      "epoch": 1.3308859520313265,
+      "grad_norm": 0.17356757819652557,
+      "learning_rate": 5.891074749862857e-06,
+      "loss": 1.5555,
+      "step": 1360
+    },
+    {
+      "epoch": 1.3700440528634361,
+      "grad_norm": 0.1258653849363327,
+      "learning_rate": 3.0203063964990617e-06,
+      "loss": 1.5775,
+      "step": 1400
+    },
+    {
+      "epoch": 1.4092021536955457,
+      "grad_norm": 0.12249883264303207,
+      "learning_rate": 1.0908391628854041e-06,
+      "loss": 1.5619,
+      "step": 1440
+    },
+    {
+      "epoch": 1.4483602545276555,
+      "grad_norm": 0.1455027014017105,
+      "learning_rate": 1.2140078057101266e-07,
+      "loss": 1.5342,
+      "step": 1480
+    },
+    {
+      "epoch": 1.4679393049437102,
+      "step": 1500,
+      "total_flos": 1.105565365842985e+17,
+      "train_loss": 1.763797264099121,
+      "train_runtime": 30390.5778,
+      "train_samples_per_second": 0.395,
+      "train_steps_per_second": 0.049
+    }
+  ],
+  "logging_steps": 40,
+  "max_steps": 1500,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 2,
+  "save_steps": 600,
+  "stateful_callbacks": {
+    "EarlyStoppingCallback": {
+      "args": {
+        "early_stopping_patience": 3,
+        "early_stopping_threshold": 0.05
+      },
+      "attributes": {
+        "early_stopping_patience_counter": 0
+      }
+    },
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 1.105565365842985e+17,
+  "train_batch_size": 2,
+  "trial_name": null,
+  "trial_params": null
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f306e4b6253715e26a88fa57ccca63669e96880d48de4c2a1e644ce5571bcc5a
+size 5880