solves hf cli error

docs/TOKEN_VALIDATION_FIX.md

@@ -0,0 +1,183 @@
+# Hugging Face Token Validation Fix
+
+## Problem Description
+
+The original launch script was using the `hf` CLI command to validate Hugging Face tokens, which was causing authentication failures even with valid tokens. This was due to:
+
+1. CLI installation issues
+2. Inconsistent token format handling
+3. Poor error reporting
+
+## Solution Implementation
+
+### New Python-Based Validation System
+
+We've implemented a robust Python-based token validation system using the official `huggingface_hub` API:
+
+#### Key Components
+
+1. **`scripts/validate_hf_token.py`** - Main validation script
+2. **Updated `launch.sh`** - Modified to use Python validation
+3. **`tests/test_token_validation.py`** - Test suite for validation
+4. **`scripts/check_dependencies.py`** - Dependency verification
+
+### Features
+
+- ✅ **Robust Error Handling**: Detailed error messages for different failure types
+- ✅ **JSON Output**: Structured responses for easy parsing
+- ✅ **Multiple Input Methods**: Command line arguments or environment variables
+- ✅ **Username Extraction**: Automatically retrieves username from valid tokens
+- ✅ **Dependency Checking**: Verifies required packages are installed
+
+## Usage
+
+### Direct Script Usage
+
+```bash
+# Using command line argument
+python scripts/validate_hf_token.py hf_your_token_here
+
+# Using environment variable
+export HF_TOKEN=hf_your_token_here
+python scripts/validate_hf_token.py
+```
+
+### Expected Output
+
+**Success:**
+```json
+{"success": true, "username": "YourUsername", "error": null}
+```
+
+**Failure:**
+```json
+{"success": false, "username": null, "error": "Invalid token - unauthorized access"}
+```
+
+### Integration with Launch Script
+
+The `launch.sh` script now automatically:
+
+1. Prompts for your HF token
+2. Validates it using the Python script
+3. Extracts your username automatically
+4. Provides detailed error messages if validation fails
+
+## Error Types and Solutions
+
+### Common Error Messages
+
+| Error Message | Cause | Solution |
+|---------------|-------|----------|
+| "Invalid token - unauthorized access" | Token is invalid or expired | Generate new token at https://huggingface.co/settings/tokens |
+| "Token lacks required permissions" | Token doesn't have write access | Ensure token has write permissions |
+| "Network error" | Connection issues | Check internet connection |
+| "Failed to run token validation script" | Missing dependencies | Run `pip install huggingface_hub` |
+
+### Dependency Installation
+
+```bash
+# Install required dependencies
+pip install huggingface_hub
+
+# Check all dependencies
+python scripts/check_dependencies.py
+
+# Install all requirements
+pip install -r requirements/requirements.txt
+```
+
+## Testing
+
+### Run the Test Suite
+
+```bash
+python tests/test_token_validation.py
+```
+
+### Manual Testing
+
+```bash
+# Test with your token
+python scripts/validate_hf_token.py hf_your_token_here
+
+# Test dependency check
+python scripts/check_dependencies.py
+```
+
+## Troubleshooting
+
+### If Token Validation Still Fails
+
+1. **Check Token Format**: Ensure token starts with `hf_`
+2. **Verify Token Permissions**: Token needs read/write access
+3. **Check Network**: Ensure internet connection is stable
+4. **Update Dependencies**: Run `pip install --upgrade huggingface_hub`
+
+### If Launch Script Fails
+
+1. **Check Python Path**: Ensure `python3` is available
+2. **Verify Script Permissions**: Script should be executable
+3. **Check JSON Parsing**: Ensure Python can parse JSON output
+4. **Review Error Messages**: Check the specific error in launch.sh output
+
+## Technical Details
+
+### Token Validation Process
+
+1. **Environment Setup**: Sets `HUGGING_FACE_HUB_TOKEN` environment variable
+2. **API Client Creation**: Initializes `HfApi()` client
+3. **User Info Retrieval**: Calls `api.whoami()` to validate token
+4. **Username Extraction**: Extracts username from user info
+5. **Error Handling**: Catches and categorizes different error types
+
+### JSON Parsing in Shell
+
+The launch script uses Python's JSON parser to safely extract values:
+
+```bash
+local success=$(echo "$result" | python3 -c "
+import sys, json
+try:
+    data = json.load(sys.stdin)
+    print(data.get('success', False))
+except:
+    print('False')
+")
+```
+
+## Migration from Old System
+
+### Before (CLI-based)
+```bash
+if hf whoami >/dev/null 2>&1; then
+    HF_USERNAME=$(hf whoami | head -n1 | tr -d '\n')
+```
+
+### After (Python-based)
+```bash
+if result=$(python3 scripts/validate_hf_token.py "$token" 2>/dev/null); then
+    # Parse JSON result with error handling
+    local success=$(echo "$result" | python3 -c "...")
+    local username=$(echo "$result" | python3 -c "...")
+```
+
+## Benefits
+
+1. **Reliability**: Uses official Python API instead of CLI
+2. **Error Reporting**: Detailed error messages for debugging
+3. **Cross-Platform**: Works on Windows, Linux, and macOS
+4. **Maintainability**: Easy to update and extend
+5. **Testing**: Comprehensive test suite included
+
+## Future Enhancements
+
+- [ ] Add token expiration checking
+- [ ] Implement token refresh functionality
+- [ ] Add support for organization tokens
+- [ ] Create GUI for token management
+- [ ] Add token security validation
+
+---
+
+**Note**: This fix ensures that valid Hugging Face tokens are properly recognized and that users get clear feedback when there are authentication issues. 

launch.sh

@@ -89,13 +89,44 @@ validate_hf_token_and_get_username() {
         return 1
     fi
     
-    # Test the token and get username
-    export HF_TOKEN="$token"
-    if hf whoami >/dev/null 2>&1; then
-        # Get username from whoami command
-        HF_USERNAME=$(hf whoami | head -n1 | tr -d '\n')
-        return 0
+    # Use Python script for validation
+    local result
+    if result=$(python3 scripts/validate_hf_token.py "$token" 2>/dev/null); then
+        # Parse JSON result using a more robust approach
+        local success=$(echo "$result" | python3 -c "
+import sys, json
+try:
+    data = json.load(sys.stdin)
+    print(data.get('success', False))
+except:
+    print('False')
+")
+        local username=$(echo "$result" | python3 -c "
+import sys, json
+try:
+    data = json.load(sys.stdin)
+    print(data.get('username', ''))
+except:
+    print('')
+")
+        local error=$(echo "$result" | python3 -c "
+import sys, json
+try:
+    data = json.load(sys.stdin)
+    print(data.get('error', 'Unknown error'))
+except:
+    print('Failed to parse response')
+")
+        
+        if [ "$success" = "True" ] && [ -n "$username" ]; then
+            HF_USERNAME="$username"
+            return 0
+        else
+            print_error "Token validation failed: $error"
+            return 1
+        fi
     else
+        print_error "Failed to run token validation script. Make sure huggingface_hub is installed."
         return 1
     fi
 }

scripts/check_dependencies.py

@@ -0,0 +1,74 @@
+#!/usr/bin/env python3
+"""
+Dependency Check Script
+
+This script checks if all required dependencies are installed for the
+SmolLM3 fine-tuning pipeline.
+"""
+
+import sys
+import importlib
+
+def check_dependency(module_name: str, package_name: str = None) -> bool:
+    """
+    Check if a Python module is available.
+    
+    Args:
+        module_name (str): The module name to check
+        package_name (str): The package name for pip installation (if different)
+        
+    Returns:
+        bool: True if module is available, False otherwise
+    """
+    try:
+        importlib.import_module(module_name)
+        return True
+    except ImportError:
+        return False
+
+def main():
+    """Check all required dependencies."""
+    
+    print("🔍 Checking dependencies for SmolLM3 Fine-tuning Pipeline")
+    print("=" * 60)
+    
+    # Required dependencies
+    dependencies = [
+        ("huggingface_hub", "huggingface_hub"),
+        ("torch", "torch"),
+        ("transformers", "transformers"),
+        ("datasets", "datasets"),
+        ("accelerate", "accelerate"),
+        ("peft", "peft"),
+        ("trl", "trl"),
+        ("bitsandbytes", "bitsandbytes"),
+    ]
+    
+    missing_deps = []
+    all_good = True
+    
+    for module_name, package_name in dependencies:
+        if check_dependency(module_name):
+            print(f"✅ {module_name}")
+        else:
+            print(f"❌ {module_name} (install with: pip install {package_name})")
+            missing_deps.append(package_name)
+            all_good = False
+    
+    print("\n" + "=" * 60)
+    
+    if all_good:
+        print("✅ All dependencies are installed!")
+        print("🚀 You're ready to run the fine-tuning pipeline!")
+    else:
+        print("❌ Missing dependencies detected!")
+        print("\nTo install missing dependencies, run:")
+        print(f"pip install {' '.join(missing_deps)}")
+        print("\nOr install all requirements:")
+        print("pip install -r requirements/requirements.txt")
+    
+    return all_good
+
+if __name__ == "__main__":
+    success = main()
+    sys.exit(0 if success else 1) 

scripts/validate_hf_token.py

@@ -0,0 +1,90 @@
+#!/usr/bin/env python3
+"""
+Hugging Face Token Validation Script
+
+This script validates a Hugging Face token and retrieves the associated username
+using the huggingface_hub Python API.
+"""
+
+import sys
+import os
+from typing import Optional, Tuple
+from huggingface_hub import HfApi, login
+import json
+
+def validate_hf_token(token: str) -> Tuple[bool, Optional[str], Optional[str]]:
+    """
+    Validate a Hugging Face token and return the username.
+    
+    Args:
+        token (str): The Hugging Face token to validate
+        
+    Returns:
+        Tuple[bool, Optional[str], Optional[str]]: 
+            - success: True if token is valid, False otherwise
+            - username: The username associated with the token (if valid)
+            - error_message: Error message if validation failed
+    """
+    try:
+        # Set the token as environment variable
+        os.environ["HUGGING_FACE_HUB_TOKEN"] = token
+        
+        # Create API client
+        api = HfApi()
+        
+        # Try to get user info - this will fail if token is invalid
+        user_info = api.whoami()
+        
+        # Extract username from user info
+        username = user_info.get("name", user_info.get("username"))
+        
+        if not username:
+            return False, None, "Could not retrieve username from token"
+            
+        return True, username, None
+        
+    except Exception as e:
+        error_msg = str(e)
+        if "401" in error_msg or "unauthorized" in error_msg.lower():
+            return False, None, "Invalid token - unauthorized access"
+        elif "403" in error_msg:
+            return False, None, "Token lacks required permissions"
+        elif "network" in error_msg.lower() or "connection" in error_msg.lower():
+            return False, None, f"Network error: {error_msg}"
+        else:
+            return False, None, f"Validation error: {error_msg}"
+
+def main():
+    """Main function to validate token from command line or environment."""
+    
+    # Get token from command line argument or environment variable
+    if len(sys.argv) > 1:
+        token = sys.argv[1]
+    else:
+        token = os.environ.get("HF_TOKEN") or os.environ.get("HUGGING_FACE_HUB_TOKEN")
+    
+    if not token:
+        print(json.dumps({
+            "success": False,
+            "username": None,
+            "error": "No token provided. Use as argument or set HF_TOKEN environment variable."
+        }))
+        sys.exit(1)
+    
+    # Validate token
+    success, username, error = validate_hf_token(token)
+    
+    # Return result as JSON for easy parsing
+    result = {
+        "success": success,
+        "username": username,
+        "error": error
+    }
+    
+    print(json.dumps(result))
+    
+    # Exit with appropriate code
+    sys.exit(0 if success else 1)
+
+if __name__ == "__main__":
+    main() 

tests/test_token_validation.py

@@ -0,0 +1,61 @@
+#!/usr/bin/env python3
+"""
+Test script for Hugging Face token validation
+"""
+
+import sys
+import os
+sys.path.append(os.path.join(os.path.dirname(__file__), '..', 'scripts'))
+
+from validate_hf_token import validate_hf_token
+
+def test_token_validation():
+    """Test the token validation function."""
+    
+    # Test with a valid token (you can replace this with your own token for testing)
+    test_token = "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"
+    
+    print("Testing token validation...")
+    print(f"Token: {test_token[:10]}...")
+    
+    success, username, error = validate_hf_token(test_token)
+    
+    if success:
+        print(f"✅ Token validation successful!")
+        print(f"Username: {username}")
+    else:
+        print(f"❌ Token validation failed: {error}")
+    
+    return success
+
+def test_invalid_token():
+    """Test with an invalid token."""
+    
+    invalid_token = "hf_invalid_token_for_testing"
+    
+    print("\nTesting invalid token...")
+    success, username, error = validate_hf_token(invalid_token)
+    
+    if not success:
+        print(f"✅ Correctly rejected invalid token: {error}")
+    else:
+        print(f"❌ Unexpectedly accepted invalid token")
+    
+    return not success
+
+if __name__ == "__main__":
+    print("🧪 Testing Hugging Face Token Validation")
+    print("=" * 50)
+    
+    # Test valid token
+    valid_result = test_token_validation()
+    
+    # Test invalid token
+    invalid_result = test_invalid_token()
+    
+    print("\n" + "=" * 50)
+    if valid_result and invalid_result:
+        print("✅ All tests passed!")
+    else:
+        print("❌ Some tests failed!")
+        sys.exit(1)