Upload 3 files

docs/AI_SBOM_API_doc.md

@@ -2,7 +2,28 @@
 
 ## Overview
 
-The AI SBOM Generator API provides a comprehensive solution for generating CycloneDX-compliant AI Bill of Materials (AI SBOM) for Hugging Face models. This document outlines the available API endpoints, their functionality, and how to interact with them using cURL commands.
+The AI SBOM Generator API provides a comprehensive solution for generating CycloneDX-compliant AI Software Bill of Materials (AI SBOM) for Hugging Face models. This API uses a configurable field registry system to extract and score AI SBOM fields across 5 categories, providing detailed completeness assessment and standards compliance.
+
+---
+
+## Table of Contents
+- [Base URL](#base-url)
+- [API Endpoints](#api-endpoints)
+  - [API Status](#api-status)
+  - [Registry Status](#registry-status)
+  - [Generate AI SBOM](#generate-ai-sbom)
+  - [Generate AI SBOM with Completeness Score Report](#generate-ai-sbom-with-completeness-score-report)
+  - [Get Completeness Score Only](#get-completeness-score-only)
+  - [Download Generated AI SBOM](#download-generated-ai-sbom)
+  - [Form-Based Generation (Web UI)](#form-based-generation-web-ui)
+- [Web UI](#web-ui)
+- [Security Features](#security-features)
+- [Field Registry System](#field-registry-system)
+- [Completeness Score](#completeness-score)
+- [Notes on Using the API](#notes-on-using-the-api)
+- [Error Handling](#error-handling)
+
+---
 
 ## Base URL
 
@@ -13,9 +34,11 @@ https://aetheris-ai-aibom-generator.hf.space
 
 Replace this with your actual deployment URL.
 
+---
+
 ## API Endpoints
 
-### Status Endpoint
+### API Status
 
 **Purpose**: Check if the API is operational and get version information.
 
@@ -37,7 +60,47 @@ curl -X GET "https://aetheris-ai-aibom-generator.hf.space/status"
 }
 ```
 
-### Generate AI SBOM Endpoint
+---
+
+### Registry Status
+
+**Purpose**: Check the field registry configuration status and available fields.
+
+**Endpoint**: `/api/registry/status`
+
+**Method**: GET
+
+**cURL Example**:
+```bash
+curl -X GET "https://aetheris-ai-aibom-generator.hf.space/api/registry/status"
+```
+
+**Expected Response**:
+```json
+{
+  "registry_available": true,
+  "total_fields": 29,
+  "categories": [
+    "required_fields",
+    "metadata", 
+    "component_basic",
+    "component_model_card",
+    "external_references"
+  ],
+  "field_count_by_category": {
+    "required_fields": 4,
+    "metadata": 5,
+    "component_basic": 5,
+    "component_model_card": 14,
+    "external_references": 1
+  },
+  "registry_manager_loaded": true
+}
+```
+
+---
+
+### Generate AI SBOM
 
 **Purpose**: Generate an AI SBOM for a specified Hugging Face model.
 
@@ -56,7 +119,7 @@ curl -X GET "https://aetheris-ai-aibom-generator.hf.space/status"
 curl -X POST "https://aetheris-ai-aibom-generator.hf.space/api/generate" \
   -H "Content-Type: application/json" \
   -d '{
-    "model_id": "meta-llama/Llama-2-7b-chat-hf",
+    "model_id": "deepseek-ai/DeepSeek-R1",
     "include_inference": true,
     "use_best_practices": true
   }'
@@ -68,65 +131,172 @@ curl -X POST "https://aetheris-ai-aibom-generator.hf.space/api/generate" \
   "aibom": {
     "bomFormat": "CycloneDX",
     "specVersion": "1.6",
-    "serialNumber": "urn:uuid:...",
-    "version": 1,
-    "metadata": { ... },
-    "components": [ ... ],
-    "dependencies": [ ... ]
+    "serialNumber": "urn:uuid:deepseek-ai-DeepSeek-R1",
+    "version": "1.0.0",
+    "metadata": {
+      "timestamp": "2025-07-15T18:31:18Z",
+      "tools": [
+        {
+          "vendor": "Aetheris AI",
+          "name": "AI SBOM Generator",
+          "version": "1.0.0"
+        }
+      ],
+      "properties": [
+        {
+          "name": "primaryPurpose",
+          "value": "text-generation"
+        },
+        {
+          "name": "suppliedBy", 
+          "value": "deepseek-ai"
+        }
+      ]
+    },
+    "components": [
+      {
+        "type": "machine-learning-model",
+        "name": "DeepSeek-R1",
+        "purl": "pkg:huggingface/deepseek-ai/DeepSeek-R1",
+        "description": "Advanced reasoning model with enhanced capabilities",
+        "licenses": [
+          {
+            "license": {
+              "name": "DeepSeek License"
+            }
+          }
+        ],
+        "modelCard": {
+          "limitation": "Model may have limitations in certain domains"
+        }
+      }
+    ],
+    "externalReferences": [
+      {
+        "type": "distribution",
+        "url": "https://huggingface.co/deepseek-ai/DeepSeek-R1"
+      }
+    ]
   },
-  "model_id": "meta-llama/Llama-2-7b-chat-hf",
-  "generated_at": "2025-04-24T20:30:00Z",
-  "request_id": "...",
-  "download_url": "/output/meta-llama_Llama-2-7b-chat-hf_....json"
+  "model_id": "deepseek-ai/DeepSeek-R1",
+  "generated_at": "2025-07-15T18:31:18Z",
+  "request_id": "550e8400-e29b-41d4-a716-446655440000",
+  "download_url": "/output/deepseek-ai_DeepSeek-R1_ai_sbom.json"
 }
 ```
 
-### Generate AI SBOM with Enhancement Report
+---
+
+### Generate AI SBOM with Completeness Score Report
 
-**Purpose**: Generate an AI SBOM with a detailed enhancement report.
+**Purpose**: Generate an AI SBOM along with a detailed completeness score report.
 
 **Endpoint**: `/api/generate-with-report`
 
 **Method**: POST
 
-**Parameters**: Same as `/api/generate`
+**Parameters**: Same as Generate AI SBOM
 
 **cURL Example**:
 ```bash
 curl -X POST "https://aetheris-ai-aibom-generator.hf.space/api/generate-with-report" \
   -H "Content-Type: application/json" \
   -d '{
-    "model_id": "meta-llama/Llama-2-7b-chat-hf",
+    "model_id": "deepseek-ai/DeepSeek-R1",
     "include_inference": true,
     "use_best_practices": true
   }'
 ```
 
-**Expected Response**: JSON containing the generated AI SBOM, model ID, timestamp, download URL, and enhancement report.
+**Expected Response**: Same as Generate AI SBOM plus completeness score details.
 ```json
 {
   "aibom": { ... },
-  "model_id": "meta-llama/Llama-2-7b-chat-hf",
-  "generated_at": "2025-04-24T20:30:00Z",
-  "request_id": "...",
-  "download_url": "/output/meta-llama_Llama-2-7b-chat-hf_....json",
-  "enhancement_report": {
-    "ai_enhanced": true,
-    "ai_model": "BERT-base-uncased",
-    "original_score": {
-      "total_score": 65.5,
-      "completeness_score": 65.5
+  "model_id": "deepseek-ai/DeepSeek-R1",
+  "generated_at": "2025-07-15T18:31:18Z",
+  "request_id": "550e8400-e29b-41d4-a716-446655440000",
+  "download_url": "/output/deepseek-ai_DeepSeek-R1_ai_sbom.json",
+  "completeness_score": {
+    "total_score": 62.3,
+    "section_scores": {
+      "required_fields": 20.0,
+      "metadata": 8.0,
+      "component_basic": 20.0,
+      "component_model_card": 4.3,
+      "external_references": 10.0
     },
-    "final_score": {
-      "total_score": 85.2,
-      "completeness_score": 85.2
+    "max_scores": {
+      "required_fields": 20,
+      "metadata": 20,
+      "component_basic": 20,
+      "component_model_card": 30,
+      "external_references": 10
     },
-    "improvement": 19.7
+    "field_checklist": {
+      "bomFormat": "present",
+      "specVersion": "present",
+      "serialNumber": "present",
+      "version": "present",
+      "primaryPurpose": "present",
+      "suppliedBy": "present",
+      "standardCompliance": "missing",
+      "domain": "missing",
+      "autonomyType": "missing",
+      "name": "present",
+      "type": "present",
+      "purl": "present",
+      "description": "present",
+      "licenses": "present",
+      "energyConsumption": "missing",
+      "hyperparameter": "missing",
+      "limitation": "present",
+      "safetyRiskAssessment": "missing",
+      "typeOfModel": "present",
+      "modelExplainability": "missing",
+      "energyQuantity": "missing",
+      "energyUnit": "missing",
+      "informationAboutTraining": "missing",
+      "informationAboutApplication": "missing",
+      "metric": "missing",
+      "metricDecisionThreshold": "missing",
+      "modelDataPreprocessing": "missing",
+      "useSensitivePersonalInformation": "missing",
+      "downloadLocation": "present"
+    },
+    "category_details": {
+      "required_fields": {
+        "present_fields": 4,
+        "total_fields": 4,
+        "percentage": 100.0
+      },
+      "metadata": {
+        "present_fields": 2,
+        "total_fields": 5,
+        "percentage": 40.0
+      },
+      "component_basic": {
+        "present_fields": 5,
+        "total_fields": 5,
+        "percentage": 100.0
+      },
+      "component_model_card": {
+        "present_fields": 2,
+        "total_fields": 14,
+        "percentage": 14.3
+      },
+      "external_references": {
+        "present_fields": 1,
+        "total_fields": 1,
+        "percentage": 100.0
+      }
+    }
   }
 }
 ```
 
-### Get Model Score
+---
+
+### Get Completeness Score Only
 
 **Purpose**: Get the completeness score for a model without generating a full AI SBOM.
 
@@ -136,24 +306,25 @@ curl -X POST "https://aetheris-ai-aibom-generator.hf.space/api/generate-with-rep
 
 **Parameters**:
 - `model_id` (path parameter): The Hugging Face model ID
-- `hf_token` (query parameter, optional): Hugging Face API token for accessing private models
 - `use_best_practices` (query parameter, optional): Whether to use industry best practices for scoring (default: true)
+- `hf_token` (query parameter, optional): Hugging Face API token for accessing private models
 
 **cURL Example**:
 ```bash
-curl -X GET "https://aetheris-ai-aibom-generator.hf.space/api/models/meta-llama/Llama-2-7b-chat-hf/score?use_best_practices=true"
+curl -X GET "https://aetheris-ai-aibom-generator.hf.space/api/models/deepseek-ai/DeepSeek-R1/score?use_best_practices=true"
 ```
 
-**Expected Response**: JSON containing the completeness score information.
+**Expected Response**:
 ```json
 {
-  "total_score": 85.2,
+  "model_id": "deepseek-ai/DeepSeek-R1",
+  "total_score": 62.3,
   "section_scores": {
-    "required_fields": 20,
-    "metadata": 18.5,
-    "component_basic": 20,
-    "component_model_card": 20.7,
-    "external_references": 6
+    "required_fields": 20.0,
+    "metadata": 8.0,
+    "component_basic": 20.0,
+    "component_model_card": 4.3,
+    "external_references": 10.0
   },
   "max_scores": {
     "required_fields": 20,
@@ -161,136 +332,205 @@ curl -X GET "https://aetheris-ai-aibom-generator.hf.space/api/models/meta-llama/
     "component_basic": 20,
     "component_model_card": 30,
     "external_references": 10
-  }
+  },
+  "field_checklist": {
+    "bomFormat": "present",
+    "specVersion": "present",
+    "name": "present",
+    "downloadLocation": "present"
+  },
+  "generated_at": "2025-07-15T18:31:18Z",
+  "request_id": "550e8400-e29b-41d4-a716-446655440000"
 }
 ```
 
+---
+
 ### Download Generated AI SBOM
 
 **Purpose**: Download a previously generated AI SBOM file.
 
-**Endpoint**: `/download/{filename}`
+**Endpoint**: `/output/{filename}`
 
 **Method**: GET
 
-**Parameters**:
-- `filename` (path parameter): The filename of the AI SBOM to download
-
 **cURL Example**:
 ```bash
-curl -X GET "https://aetheris-ai-aibom-generator.hf.space/download/{filename}" \
-  -o "downloaded_aibom.json"
+curl -X GET "https://aetheris-ai-aibom-generator.hf.space/output/deepseek-ai_DeepSeek-R1_ai_sbom.json" \
+  -o "deepseek_r1_aibom.json"
 ```
 
-**Expected Response**: The AI SBOM JSON file will be downloaded to your local machine.
+---
 
 ### Form-Based Generation (Web UI)
 
-**Purpose**: Generate an AI SBOM using form data (typically used by the web UI).
+**Purpose**: Generate AI SBOM through the web interface form submission.
 
 **Endpoint**: `/generate`
 
 **Method**: POST
 
+**Content-Type**: `application/x-www-form-urlencoded`
+
 **Parameters**:
-- `model_id` (form field, required): The Hugging Face model ID
-- `include_inference` (form field, optional): Whether to use AI inference to enhance the AI SBOM
-- `use_best_practices` (form field, optional): Whether to use industry best practices for scoring
+- `model_id` (required): The Hugging Face model ID
+- `g-recaptcha-response` (required): reCAPTCHA response token
 
 **cURL Example**:
 ```bash
 curl -X POST "https://aetheris-ai-aibom-generator.hf.space/generate" \
-  -F "model_id=meta-llama/Llama-2-7b-chat-hf" \
-  -F "include_inference=true" \
-  -F "use_best_practices=true"
+  -H "Content-Type: application/x-www-form-urlencoded" \
+  -d "model_id=deepseek-ai/DeepSeek-R1&g-recaptcha-response=YOUR_RECAPTCHA_TOKEN"
 ```
 
-**Expected Response**: HTML page with the generated AI SBOM results.
+---
 
 ## Web UI
 
-The API also provides a web user interface for generating AI SBOMs without writing code:
+The API also provides a user-friendly web interface accessible at the base URL. The web UI includes:
 
-**URL**: `https://aetheris-ai-aibom-generator.hf.space/`
+- **Model ID input field** with validation
+- **reCAPTCHA protection** against automated abuse
+- **Real-time generation** with progress indicators
+- **Downloadable results** with completeness scoring
+- **Field checklist visualization** showing extraction results
+- **Category-based scoring breakdown**
 
-The web UI allows you to:
-1. Enter a Hugging Face model ID
-2. Configure generation options
-3. Generate an AI SBOM
-4. View the results in a human-friendly format
-5. Download the generated AI SBOM as a JSON file
+---
 
-## Understanding the Field Checklist
+## Security Features
 
-In the Field Checklist tab of the results page, you'll see a list of fields with check marks (✔/✘) and stars (★). Here's what they mean:
+### Rate Limiting
+- **10 requests per minute** per IP address
+- **5 concurrent requests** maximum
+- **1MB request size limit**
 
-- **Check marks**:
-  - ✔: Field is present in the AI SBOM
-  - ✘: Field is missing from the AI SBOM
+### reCAPTCHA Protection
+- **Google reCAPTCHA v2** integration for web UI
+- **Automated bot detection** and prevention
+- **Configurable through environment variables**
 
-- **Stars** (importance level):
-  - ★★★ (three stars): Critical fields - Essential for a valid and complete AI SBOM
-  - ★★ (two stars): Important fields - Valuable information that enhances completeness
-  - ★ (one star): Supplementary fields - Additional context and details (optional)
+### Input Validation
+- **Model ID format validation** (alphanumeric, hyphens, underscores, forward slashes)
+- **XSS protection** through HTML escaping
+- **SQL injection prevention** through parameterized queries
 
-## Security Features
+---
+
+## Field Registry System
+
+The AI SBOM Generator uses a configurable field registry system that enables:
+
+### **29 Configurable Fields** across 5 categories:
+- **Required Fields (4)**: bomFormat, specVersion, serialNumber, version
+- **Metadata (5)**: primaryPurpose, suppliedBy, standardCompliance, domain, autonomyType  
+- **Component Basic (5)**: name, type, purl, description, licenses
+- **Component Model Card (14)**: energyConsumption, hyperparameter, limitation, safetyRiskAssessment, typeOfModel, modelExplainability, energyQuantity, energyUnit, informationAboutTraining, informationAboutApplication, metric, metricDecisionThreshold, modelDataPreprocessing, useSensitivePersonalInformation
+- **External References (1)**: downloadLocation
 
-The API includes several security features to protect against Denial of Service (DoS) attacks:
+### **Multi-Strategy Extraction**:
+1. **HuggingFace API** → Direct metadata extraction (High confidence)
+2. **Model Card** → Structured documentation parsing (Medium-high confidence)
+3. **Config Files** → Technical details from JSON files (High confidence)
+4. **Text Patterns** → Regex extraction from README (Medium confidence)
+5. **Intelligent Inference** → Smart defaults from context (Medium confidence)
+6. **Fallback Values** → Placeholders when no data available (Low confidence)
 
-1. **Rate Limiting**: Limits the number of requests a single IP address can make within a specific time window.
+### **SPDX 3.0 Compatibility**:
+- **100% field coverage** with SPDX 3.0 AI Profile specification
+- **59% exact field name matches** with official SPDX 3.0 fields
+- **Future dual-format support** for both CycloneDX and SPDX output
+- **Current limitation** does not generate output in SPDX format
 
-2. **Concurrency Limiting**: Restricts the total number of simultaneous requests being processed to prevent resource exhaustion.
+---
 
-3. **Request Size Limiting**: Prevents attackers from sending extremely large payloads that could consume memory or processing resources.
+## Completeness Score
+
+The completeness score is calculated using a weighted scoring system across five categories:
+
+### **Scoring Categories**:
+- **Required Fields (20%)**: Essential CycloneDX infrastructure
+- **Metadata (20%)**: AI-specific metadata and provenance  
+- **Component Basic (20%)**: Core component identification
+- **Component Model Card (30%)**: Advanced AI model documentation
+- **External References (10%)**: Distribution and reference links
+
+### **Field Tiers**:
+- **Critical (C)**: Essential fields with 3x weight multiplier
+- **Important (I)**: Valuable fields with 2x weight multiplier
+- **Supplementary (S)**: Additional fields with 1x weight multiplier
 
-4. **API Key Authentication** (optional): When configured, requires an API key for accessing API endpoints, enabling tracking and control of API usage.
+### **Score Interpretation**:
+- **90-100**: Exceptional documentation quality
+- **80-89**: Comprehensive documentation
+- **70-79**: Good documentation with minor gaps
+- **60-69**: Adequate documentation with some missing elements
+- **50-59**: Basic documentation with significant gaps
+- **Below 50**: Insufficient documentation
 
-5. **CAPTCHA Verification** (optional): When configured for the web interface, helps ensure requests come from humans rather than bots.
+### **Confidence-Based Filtering**:
+- Only fields extracted with **medium** or **high** confidence contribute to the score
+- **Low** or **none** confidence extractions are excluded to ensure score reliability
+- Individual field failures don't prevent overall SBOM generation
+
+---
 
 ## Notes on Using the API
 
-1. When deployed on Hugging Face Spaces, use the correct URL format as shown in the examples.
-2. Some endpoints may have rate limiting or require authentication.
-3. For large responses, consider adding appropriate timeout settings in your requests.
-4. If you encounter CORS issues, you may need to add appropriate headers.
-5. For downloading files, specify the output file name in your client code.
+### **Model ID Format**
+- Use the exact Hugging Face model identifier (e.g., `meta-llama/Llama-2-7b-chat-hf`)
+- Model IDs are case-sensitive
+- Private models require a valid `hf_token`
+
+### **Response Times**
+- **Simple models**: 5-15 seconds
+- **Complex models with inference**: 30-60 seconds
+- **Large models**: Up to 2 minutes
+
+### **File Storage**
+- Generated AI SBOMs are stored temporarily (7 days)
+- Download URLs are valid for the file retention period
+- Files are automatically cleaned up to manage storage
+
+### **Best Practices**
+- Use `use_best_practices=true` for industry-standard scoring
+- Include `include_inference=true` for enhanced field extraction
+- Cache results locally to avoid repeated API calls for the same model
+- Use the registry status endpoint to verify system configuration
+
+---
 
 ## Error Handling
 
-The API returns standard HTTP status codes:
-- 200: Success
-- 400: Bad Request (invalid parameters)
-- 404: Not Found (resource not found)
-- 429: Too Many Requests (rate limit exceeded)
-- 500: Internal Server Error (server-side error)
-- 503: Service Unavailable (server at capacity)
+### **Common HTTP Status Codes**:
+- **200 OK**: Successful request
+- **400 Bad Request**: Invalid model ID format or missing parameters
+- **404 Not Found**: Model not found on Hugging Face
+- **429 Too Many Requests**: Rate limit exceeded
+- **500 Internal Server Error**: Server-side processing error
 
-Error responses include a detail message explaining the error:
+### **Error Response Format**:
 ```json
 {
-  "detail": "Error generating AI SBOM: Model not found"
+  "detail": "Error description",
+  "error_code": "SPECIFIC_ERROR_CODE",
+  "timestamp": "2025-07-15T18:31:18Z"
 }
 ```
 
-## Completeness Score
-
-The completeness score is calculated based on the presence and quality of various fields in the AI SBOM. The score is broken down into sections:
-
-1. **Required Fields** (20 points): Basic required fields for a valid AI SBOM
-2. **Metadata** (20 points): Information about the AI SBOM itself
-3. **Component Basic Info** (20 points): Basic information about the AI model component
-4. **Model Card** (30 points): Detailed model card information
-5. **External References** (10 points): Links to external resources
-
-The total score is a weighted sum of these section scores, with a maximum of 100 points.
+### **Common Error Scenarios**:
+- **Invalid Model ID**: Check format and existence on Hugging Face
+- **Private Model Access**: Ensure valid `hf_token` is provided
+- **Rate Limiting**: Wait before retrying or implement exponential backoff
+- **Registry Unavailable**: System falls back to basic field extraction
 
-## Enhancement Report
+---
 
-When AI enhancement is enabled, the API uses an inference model to extract additional information from the model card and other sources. The enhancement report shows:
+## Support and Documentation
 
-1. **Original Score**: The completeness score before enhancement
-2. **Enhanced Score**: The completeness score after enhancement
-3. **Improvement**: The point increase from enhancement
-4. **AI Model Used**: The model used for enhancement
+For additional support, documentation updates, or feature requests:
+- **GitHub Repository**: [[Link to GitHub Isuses](https://github.com/aetheris-ai/aibom-generator/issues)]
+- **API Status Page**: Use `/status` and `/api/registry/status` endpoints
+- **Web Interface**: Available at the base URL for interactive testing
 
-This helps you understand how much the AI enhancement improved the AI SBOM's completeness.
+This API provides comprehensive AI SBOM generation capabilities with industry-leading field coverage, standards compliance, and configurable scoring systems.

docs/AI_SBOM_Fields_Mapping_Reference.md

@@ -0,0 +1,204 @@
+# AI SBOM Fields Mapping Reference
+
+## Table of Contents
+
+- [Overview](#overview)
+- [Legend](#legend)
+- [Field Categories](#field-categories)
+  - [Required Fields Category](#required-fields-category)
+  - [Metadata Category](#metadata-category)
+  - [Component Basic Category](#component-basic-category)
+  - [Component Model Card Category](#component-model-card-category)
+  - [External References Category](#external-references-category)
+- [Scoring Summary](#scoring-summary)
+- [Field Extraction Strategies](#field-extraction-strategies)
+- [Standards Compatibility](#standards-compatibility)
+- [Usage Notes](#usage-notes)
+
+---
+
+## Overview
+
+This document provides a comprehensive mapping of all 29 fields used in the AI SBOM Generator, organized by category to match the UI structure. Each field includes its CycloneDX 1.6 location, scoring weight, tier classification, and description. SPDX 3.0 compatibility information is included for reference.
+
+The AI SBOM Generator uses a configurable field registry to extract, validate, and score AI model documentation across multiple sources, providing comprehensive Bill of Materials for AI systems.
+
+---
+
+## Legend
+
+### Tiers
+- **C**: Critical - Essential fields (weight: 3x, 4.0-10.0 points)
+- **I**: Important - Valuable fields (weight: 2x, 2.0-3.0 points)
+- **S**: Supplementary - Additional fields (weight: 1x, 1.0-2.0 points)
+
+### SPDX 3.0 Alignment Status (AS)
+- **🎯**: Exact Match - Matched field name and type
+- **✅**: Standard Field - Core SPDX compatibility
+- **🔄**: Semantic Match - Same concept, different name
+
+---
+
+## Field Categories
+
+### Required Fields Category
+
+Essential CycloneDX infrastructure fields that form the foundation of every SBOM document. These fields are mandatory for proper SBOM identification and compliance.
+
+| # | Field Name | CycloneDX Location | SPDX 3.0 Equivalent | Tier | AS | Points | Description |
+|---|------------|-------------------|---------------------|------|--------|--------|-------------|
+| 1 | **bomFormat** | `$.bomFormat` | Core SPDX field | C | ✅ | 4.0 | Format identifier for the SBOM (always "CycloneDX") |
+| 2 | **specVersion** | `$.specVersion` | `spdxVersion` | C | ✅ | 4.0 | CycloneDX specification version (e.g., "1.6") |
+| 3 | **serialNumber** | `$.serialNumber` | `spdxId` | C | ✅ | 4.0 | Unique identifier for this SBOM instance |
+| 4 | **version** | `$.version` | `releaseTime` | C | ✅ | 4.0 | Version of this SBOM document |
+
+**Category Result:** 4/4 fields • **20.0/20 points** • **100% weight**
+
+---
+
+### Metadata Category
+
+AI-specific metadata and provenance information that provides context about the model's purpose, supply chain, and compliance. These fields help establish the model's intended use and regulatory context.
+
+| # | Field Name | CycloneDX Location | SPDX 3.0 Equivalent | Tier | AS | Points | Description |
+|---|------------|-------------------|---------------------|------|--------|--------|-------------|
+| 5 | **primaryPurpose** | `$.metadata.properties[name="primaryPurpose"]` | `ai_intendedUse` | C | 🔄 | 4.0 | Primary intended use of the AI model |
+| 6 | **suppliedBy** | `$.metadata.properties[name="suppliedBy"]` | `supplier` | C | ✅ | 4.0 | Organization or individual who supplied the model |
+| 7 | **standardCompliance** | `$.metadata.properties[name="standardCompliance"]` | `ai_standardCompliance` | S | 🎯 | 1.0 | Compliance with AI/ML standards and regulations |
+| 8 | **domain** | `$.metadata.properties[name="domain"]` | `ai_domain` | S | 🎯 | 1.0 | Application domain or industry vertical |
+| 9 | **autonomyType** | `$.metadata.properties[name="autonomyType"]` | `ai_autonomyType` | S | 🎯 | 1.0 | Level of autonomy in decision-making |
+
+**Category Result:** 5/5 fields • **20.0/20 points** • **100% weight**
+
+---
+
+### Component Basic Category
+
+Core component identification and description fields that define the essential characteristics of the AI model. These fields provide fundamental information needed for model identification and basic documentation.
+
+| # | Field Name | CycloneDX Location | SPDX 3.0 Equivalent | Tier | AS | Points | Description |
+|---|------------|-------------------|---------------------|------|--------|--------|-------------|
+| 10 | **name** | `$.components[0].name` | `name` | C | ✅ | 4.0 | Human-readable name of the model |
+| 11 | **type** | `$.components[0].type` | `ai_AIPackage` type | I | ✅ | 2.0 | Component type (always "machine-learning-model") |
+| 12 | **purl** | `$.components[0].purl` | `externalRefs[type="purl"]` | I | ✅ | 2.0 | Package URL for unique identification |
+| 13 | **description** | `$.components[0].description` | `summary` | I | ✅ | 2.0 | Brief description of the model's purpose |
+| 14 | **licenses** | `$.components[0].licenses` | `licenseConcluded` | I | ✅ | 2.0 | License information for the model |
+
+**Category Result:** 5/5 fields • **20.0/20 points** • **100% weight**
+
+---
+
+### Component Model Card Category
+
+Advanced AI model documentation fields that provide detailed information about model characteristics, training, performance, and usage considerations. This category represents the most comprehensive AI-specific documentation.
+
+| # | Field Name | CycloneDX Location | SPDX 3.0 Equivalent | Tier | AS | Points | Description |
+|---|------------|-------------------|---------------------|------|--------|--------|-------------|
+| 15 | **energyConsumption** | `$.components[0].modelCard.properties[name="energyConsumption"]` | `ai_energyConsumption` | I | 🎯 | 2.0 | Energy consumption information |
+| 16 | **hyperparameter** | `$.components[0].modelCard.properties[name="hyperparameter"]` | `ai_hyperparameter` | I | 🎯 | 2.0 | Key hyperparameters used in training |
+| 17 | **limitation** | `$.components[0].modelCard.limitation` | `ai_limitation` | I | 🎯 | 2.0 | Known limitations and constraints |
+| 18 | **safetyRiskAssessment** | `$.components[0].modelCard.properties[name="safetyRiskAssessment"]` | `ai_safetyRiskAssessment` | I | 🎯 | 2.0 | Safety and risk assessment information |
+| 19 | **typeOfModel** | `$.metadata.properties[name="typeOfModel"]` | `ai_typeOfModel` | I | 🎯 | 2.0 | Technical classification of the model type |
+| 20 | **modelExplainability** | `$.components[0].modelCard.properties[name="modelExplainability"]` | `ai_modelExplainability` | S | 🎯 | 1.0 | Information about model interpretability |
+| 21 | **energyQuantity** | `$.components[0].modelCard.properties[name="energyQuantity"]` | `ai_energyQuantity` | S | 🎯 | 1.0 | Quantitative energy consumption metrics |
+| 22 | **energyUnit** | `$.components[0].modelCard.properties[name="energyUnit"]` | `ai_energyUnit` | S | 🎯 | 1.0 | Units for energy consumption measurements |
+| 23 | **informationAboutTraining** | `$.components[0].modelCard.properties[name="informationAboutTraining"]` | `ai_informationAboutTraining` | S | 🎯 | 1.0 | Details about the training process |
+| 24 | **informationAboutApplication** | `$.components[0].modelCard.properties[name="informationAboutApplication"]` | `ai_informationAboutApplication` | S | 🎯 | 1.0 | Information about intended applications |
+| 25 | **metric** | `$.components[0].modelCard.properties[name="metric"]` | `ai_metric` | S | 🎯 | 1.0 | Performance metrics and evaluation results |
+| 26 | **metricDecisionThreshold** | `$.components[0].modelCard.properties[name="metricDecisionThreshold"]` | `ai_metricDecisionThreshold` | S | 🎯 | 1.0 | Decision thresholds for model outputs |
+| 27 | **modelDataPreprocessing** | `$.components[0].modelCard.properties[name="modelDataPreprocessing"]` | `ai_modelDataPreprocessing` | S | 🎯 | 1.0 | Data preprocessing and preparation steps |
+| 28 | **useSensitivePersonalInformation** | `$.components[0].modelCard.properties[name="useSensitivePersonalInformation"]` | `ai_useSensitivePersonalInformation` | S | 🎯 | 1.0 | Information about sensitive data usage |
+
+**Category Result:** 14/14 fields • **30.0/30 points** • **100% weight**
+
+---
+
+### External References Category
+
+Links and distribution information that provide access to the model and related resources. These fields enable model discovery and access.
+
+| # | Field Name | CycloneDX Location | SPDX 3.0 Equivalent | Tier | AS | Points | Description |
+|---|------------|-------------------|---------------------|------|--------|--------|-------------|
+| 29 | **downloadLocation** | `$.externalReferences[type="distribution"]` | `downloadLocation` | C | ✅ | 10.0 | Primary location to download the model |
+
+**Category Result:** 1/1 fields • **10.0/10 points** • **100% weight**
+
+---
+
+## Scoring Summary
+
+The AI SBOM Generator uses a weighted scoring system to assess documentation completeness across five categories:
+
+| Category | Fields | Max Points | Weight | Description |
+|----------|--------|------------|--------|-------------|
+| **Required Fields** | 4 | 20.0 | 20% | Essential CycloneDX infrastructure |
+| **Metadata** | 5 | 20.0 | 20% | AI-specific metadata and provenance |
+| **Component Basic** | 5 | 20.0 | 20% | Core component identification |
+| **Component Model Card** | 14 | 30.0 | 30% | Advanced AI model documentation |
+| **External References** | 1 | 10.0 | 10% | Distribution and reference links |
+| **TOTAL** | **29** | **100.0** | **100%** | Maximum possible completeness score |
+
+### Tier Impact on Scoring
+- **Critical fields** (C) have 3x weight multiplier and significantly impact scoring
+- **Important fields** (I) have 2x weight multiplier and enhance documentation quality  
+- **Supplementary fields** (S) have 1x weight multiplier and provide additional context
+
+---
+
+## Field Extraction Strategies
+
+The AI SBOM Generator employs a multi-strategy extraction approach for each field, attempting extraction in the following priority order:
+
+1. **HuggingFace API** → Direct metadata extraction (High confidence)
+2. **Model Card** → Structured documentation parsing (Medium-high confidence)  
+3. **Config Files** → Technical details from JSON files (High confidence)
+4. **Text Patterns** → Regex extraction from README (Medium confidence)
+5. **Intelligent Inference** → Smart defaults from context (Medium confidence)
+6. **Fallback Values** → Placeholders when no data available (Low/no confidence)
+
+This multi-strategy approach ensures maximum field coverage while maintaining confidence scoring for each extracted value.
+
+---
+
+## Standards Compatibility
+
+### CycloneDX 1.6 (Primary Format)
+- **Primary structure** follows CycloneDX 1.6 specification
+- **Model Card extension** provides AI-specific documentation
+- **Properties mechanism** allows flexible field addition
+- **JSON Schema validation** ensures structural compliance
+
+### SPDX 3.0 AI Profile (Reference Compatibility)
+- **100% field coverage** with official SPDX 3.0 AI Profile specification
+- **17/29 fields (59%)** have exact field name matches
+- **Compatible data types** aligned with SPDX type system
+- **Future dual-format support** enables SPDX 3.0 output
+
+### Interoperability
+- **Standards-compliant output** can be converted between formats
+- **AI field preservation** maintains semantic meaning across standards
+- **Tool compatibility** with both CycloneDX and SPDX ecosystems
+
+---
+
+## Usage Notes
+
+### Configuration and Customization
+- **Registry-driven extraction**: All fields are configurable via JSON registry
+- **Scoring weights**: Adjustable per field and category
+- **Tier assignments**: Customizable based on use case requirements
+- **Extraction strategies**: Configurable priority and methods
+
+### Field Addition and Modification
+- **New fields**: Can be added to registry without code changes
+- **Weight adjustments**: Modify scoring impact through configuration
+- **Category organization**: Fields can be reorganized by category
+- **Validation rules**: Configurable per field
+
+### Performance Characteristics
+- **Automatic field discovery**: System attempts extraction for all registry fields
+- **Graceful degradation**: Individual field failures don't stop overall extraction
+- **Confidence scoring**: Each field extraction includes confidence assessment
+- **Comprehensive logging**: Detailed extraction results for debugging
+
+This comprehensive field mapping serves as the definitive reference for the AI SBOM Generator's field extraction, scoring, and documentation capabilities, with full standards compatibility for future interoperability.

docs/AI_SBOM_Generator_System_Architecture.md

@@ -0,0 +1,423 @@
+# AI SBOM Generator System Architecture
+
+## Overview
+
+The AI SBOM Generator is a configurable system that automatically generates Software Bill of Materials (SBOM) documents for AI models hosted on HuggingFace. The system uses a registry-driven architecture that allows for dynamic field configuration without code changes.
+
+## System Architecture
+
+### Core Components
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                    AI SBOM Generator                   │
+├─────────────────────────────────────────────────────────────┤
+│  Web Interface (FastAPI + HTML Templates)              │
+├─────────────────────────────────────────────────────────────┤
+│  API Layer                                             │
+│  ├── Generation Endpoints                              │
+│  ├── Scoring Endpoints                                 │
+│  └── Batch Processing                                  │
+├─────────────────────────────────────────────────────────────┤
+│  Core Generation Engine                                │
+│  ├── AIBOMGenerator (generator.py)                     │
+│  ├── Enhanced Extractor (enhanced_extractor.py)        │
+│  └── Field Registry Manager (field_registry_manager.py)│
+├─────────────────────────────────────────────────────────────┤
+│  Configuration Layer                                   │
+│  ├── Field Registry (field_registry.json)              │
+│  ├── Scoring Configuration                             │
+│  └── AIBOM Generation Rules                            │
+├─────────────────────────────────────────────────────────────┤
+│  Data Sources                                          │
+│  ├── HuggingFace API                                   │
+│  ├── Model Cards                                       │
+│  ├── Configuration Files                               │
+│  └── README Content                                    │
+└─────────────────────────────────────────────────────────────┘
+```
+
+### Key Features
+
+- **Registry-Driven Configuration**: All fields and scoring rules defined in JSON
+- **Multi-Strategy Extraction**: 6 different extraction methods per field
+- **Standards Compliance**: CycloneDX 1.6 compatible output
+- **Configurable Scoring**: Weighted scoring system with tier-based multipliers
+- **Automatic Field Discovery**: New fields added to registry are automatically processed
+- **Comprehensive Logging**: Detailed extraction and scoring logs for debugging
+
+## Process Workflow
+
+### 1. System Initialization
+
+```
+System Initialization Process:
+
+    ┌───────────────────┐
+    │  System Startup  │
+    └─────────┬─────────┘
+              │
+              ▼
+    ┌───────────────────┐
+    │ Load Field       │
+    │ Registry         │
+    └─────────┬─────────┘
+              │
+              ▼
+    ┌───────────────────┐
+    │ Initialize       │
+    │ Registry Manager │
+    └─────────┬─────────┘
+              │
+              ▼
+    ┌─────────────────┐
+    │ Load Scoring   │
+    │ Configuration  │
+    └─────────┬───────┘
+              │
+              ▼
+    ┌─────────────────┐
+    │ Initialize     │
+    │ Enhanced       │
+    │ Extractor      │
+    └─────────┬───────┘
+              │
+              ▼
+    ┌─────────────────┐
+    │  System Ready  │
+    └─────────────────┘
+```
+
+**Steps:**
+1. **Load Field Registry**: Read `field_registry.json` containing all field definitions
+2. **Initialize Registry Manager**: Create manager instance with loaded configuration
+3. **Load Scoring Configuration**: Parse scoring weights, tiers, and category definitions
+4. **Initialize Enhanced Extractor**: Create extractor with registry-driven field discovery
+5. **System Ready**: All components initialized and ready for SBOM generation
+
+### 2. SBOM Generation Process
+
+```
+SBOM Generation Workflow:
+
+User Request ──┐
+               │
+               ▼
+    ┌───────────────────┐      ┌────────────────────┐     ┌──────────────────┐
+    │ Validate Model  │─────▶│ Fetch Model Info  │───▶│ Initialize      │
+    │ ID              │      │                   │    │ Enhanced        │
+    └───────────────────┘      └────────────────────┘    │ Extractor       │
+                                                      └──────────┬───────┘
+                                                                │
+    ┌───────────────────┐     ┌──────────────────┐                 │
+    │ Return SBOM +    │◀───│ Calculate       │◀────────────────┘
+    │ Score            │    │ Completeness    │
+    └───────────────────┘     │ Score           │
+                            └──────────────────┘
+                                    ▲
+                                    │
+                           ┌────────────────────┐
+                           │ Generate AIBOM    │
+                           │ Structure         │
+                           └────────────────────┘
+                                    ▲
+                                    │
+                           ┌────────────────────┐
+                           │ Multi-Strategy    │
+                           │ Field Processing  │
+                           └────────────────────┘
+                                    ▲
+                                    │
+                           ┌────────────────────┐
+                           │ Registry-Driven   │
+                           │ Extraction        │
+                           └────────────────────┘
+```
+
+#### 2.1 Model Information Gathering
+
+**Input**: HuggingFace model ID (e.g., `deepseek-ai/DeepSeek-R1`)
+
+**Process**:
+1. **Validate Model ID**: Check format and accessibility
+2. **Fetch Model Info**: Retrieve metadata from HuggingFace API
+3. **Download Model Card**: Get structured model documentation
+4. **Fetch Configuration Files**: Download `config.json`, `tokenizer_config.json`
+5. **Extract README Content**: Parse model description and documentation
+
+#### 2.2 Registry-Driven Field Extraction
+
+**For each of the 29 registry fields:**
+
+```
+Multi-Strategy Field Extraction:
+
+Field from Registry
+        │
+        ▼
+┌───────────────────┐     Success?
+│ Strategy 1:      │────────┐
+│ HuggingFace API  │        │
+└───────────────────┘        │
+        │                  │
+        │ Failure          │
+        ▼                  │
+┌───────────────────┐        │
+│ Strategy 2:      │        │
+│ Model Card       │        │
+└───────────────────┘        │
+        │                  │
+        │ Failure          │
+        ▼                  │
+┌───────────────────┐        │
+│ Strategy 3:      │        │
+│ Config Files     │        │
+└───────────────────┘        │
+        │                  │
+        │ Failure          │
+        ▼                  │
+┌───────────────────┐        │
+│ Strategy 4:      │        │
+│ Text Patterns    │        │
+└───────────────────┘        │
+        │                  │
+        │ Failure          │
+        ▼                  │
+┌───────────────────┐        │
+│ Strategy 5:      │        │
+│ Intelligent      │        │
+│ Inference        │        │
+└───────────────────┘        │
+        │                  │
+        │ Failure          │
+        ▼                  │
+┌───────────────────┐        │
+│ Strategy 6:      │        │
+│ Fallback Value   │        │
+└───────────────────┘        │
+        │                  │
+        ▼                  │
+┌───────────────────┐◀───────┘
+│ Store Result &   │
+│ Log Outcome      │
+└───────────────────┘
+```
+
+**Extraction Strategies**:
+
+1. **HuggingFace API Extraction**
+   - Direct field mapping from API response
+   - High confidence, structured data
+   - Fields: `name`, `author`, `license`, `tags`, etc.
+
+2. **Model Card Extraction**
+   - Parse structured model card YAML/metadata
+   - Medium-high confidence
+   - Fields: `limitation`, `metrics`, `datasets`, etc.
+
+3. **Configuration File Extraction**
+   - Mine technical details from config files
+   - High confidence for technical fields
+   - Fields: `typeOfModel`, `hyperparameter`, etc.
+
+4. **Text Pattern Extraction**
+   - Regex-based extraction from README content
+   - Medium confidence, requires validation
+   - Fields: `safetyRiskAssessment`, `informationAboutTraining`, etc.
+
+5. **Intelligent Inference**
+   - Smart defaults based on model characteristics
+   - Medium confidence, contextual
+   - Fields: `primaryPurpose`, `domain`, etc.
+
+6. **Fallback Values**
+   - Placeholder values when no data available
+   - Low/no confidence, maintains structure
+   - Ensures complete SBOM structure
+
+#### 2.3 AIBOM Structure Generation
+
+**Process**:
+1. **Create Base Structure**: Initialize CycloneDX 1.6 compliant structure
+2. **Populate Metadata Section**: Add extracted metadata fields
+3. **Build Component Section**: Create model component with extracted data
+4. **Add Model Card**: Include AI-specific model card information
+5. **Generate External References**: Add distribution and repository links
+6. **Create Dependencies**: Define model dependencies and relationships
+7. **Validate Structure**: Ensure CycloneDX compliance
+
+**Output Structure**:
+```json
+{
+  "bomFormat": "CycloneDX",
+  "specVersion": "1.6",
+  "serialNumber": "urn:uuid:...",
+  "version": 1,
+  "metadata": {
+    "timestamp": "...",
+    "tools": [...],
+    "component": {...},
+    "properties": [...]
+  },
+  "components": [{
+    "type": "machine-learning-model",
+    "name": "...",
+    "modelCard": {...},
+    "properties": [...]
+  }],
+  "externalReferences": [...],
+  "dependencies": [...]
+}
+```
+
+### 3. Completeness Scoring Process
+
+```
+Completeness Scoring Process:
+
+┌───────────────────┐
+│ Extracted Fields │
+└─────────┬─────────┘
+          │
+          ▼
+┌───────────────────┐
+│ Categorize       │
+│ Fields           │
+└─────────┬─────────┘
+          │
+          ▼
+┌───────────────────┐
+│ Apply Tier       │
+│ Weights          │
+│ • Critical: 3x   │
+│ • Important: 2x  │
+│ • Supplement: 1x │
+└─────────┬─────────┘
+          │
+          ▼
+┌───────────────────┐
+│ Calculate        │
+│ Category Scores  │
+│ • Required: 20   │
+│ • Metadata: 20   │
+│ • Basic: 20      │
+│ • ModelCard: 30  │
+│ • ExtRefs: 10    │
+└─────────┬─────────┘
+          │
+          ▼
+┌───────────────────┐
+│ Sum Weighted     │
+│ Scores           │
+│ (Max: 100)       │
+└─────────┬─────────┘
+          │
+          ▼
+┌───────────────────┐
+│ Generate Score   │
+│ Report           │
+└───────────────────┘
+```
+
+**Scoring Algorithm**:
+
+1. **Field Categorization**: Group fields by category (required_fields, metadata, etc.)
+2. **Tier Weight Application**: Apply multipliers (Critical: 3x, Important: 2x, Supplementary: 1x)
+3. **Category Score Calculation**: `(Fields Present / Total Fields) × Category Weight`
+4. **Final Score**: Sum of all category scores (max 100)
+
+**Category Weights**:
+- Required Fields: 20 points
+- Metadata: 20 points  
+- Component Basic: 20 points
+- Component Model Card: 30 points
+- External References: 10 points
+
+### 4. Output Generation
+
+**Generated Artifacts**:
+1. **AIBOM JSON**: CycloneDX 1.6 compliant SBOM document
+2. **Completeness Score**: Numerical score (0-100) with breakdown
+3. **Field Checklist**: Detailed field-by-field analysis
+4. **Extraction Report**: Confidence levels and data sources
+5. **Validation Results**: Compliance and quality checks
+
+## Configuration Management
+
+### Field Registry Structure
+
+The system is driven by `field_registry.json` which defines:
+
+- **Field Definitions**: All 29 extractable fields
+- **Scoring Configuration**: Weights, tiers, and categories
+- **AIBOM Generation Rules**: Structure and validation rules
+- **Extraction Strategies**: How each field should be extracted
+
+### Dynamic Configuration
+
+**Adding New Fields**:
+1. Add field definition to `field_registry.json`
+2. System automatically discovers and attempts extraction
+3. No code changes required
+
+**Updating Scoring**:
+1. Modify weights in registry configuration
+2. Changes take effect immediately
+3. Consistent scoring across all models
+
+## Quality Assurance
+
+### Validation Layers
+
+1. **Input Validation**: Model ID format and accessibility
+2. **Extraction Validation**: Data type and format checking
+3. **Structure Validation**: CycloneDX schema compliance
+4. **Scoring Validation**: Mathematical correctness
+5. **Output Validation**: JSON schema and completeness
+
+### Error Handling
+
+- **Individual Field Failures**: Don't stop overall processing
+- **Graceful Degradation**: Fallback to lower-confidence strategies
+- **Comprehensive Logging**: Detailed error tracking and debugging
+- **Recovery Mechanisms**: Automatic retry and alternative approaches
+
+## Performance Characteristics
+
+### Typical Processing Times
+
+- **Single Model**: 2-5 seconds
+- **Batch Processing**: 10-50 models/minute
+- **Registry Loading**: <1 second
+- **Field Extraction**: 1-3 seconds per model
+
+### Scalability Features
+
+- **Concurrent Processing**: Multiple models processed simultaneously
+- **Caching**: Model metadata and configuration caching
+- **Rate Limiting**: Respectful API usage
+- **Resource Management**: Memory and connection pooling
+
+## Integration Points
+
+### APIs
+
+- **Generation API**: `/api/generate` - Single model AI SBOM generation, with download URL
+- **Generation with Completness Score Report API**: `/api/generate-with-report` - Generation API with completness scoring report
+- **Completness Score Report Only API**: `/api/models/{model_id}/score` - Get the completeness score for a model without generating AI SBOM
+
+### Data Sources
+
+- **HuggingFace Hub**: Primary model metadata source
+- **Model Repositories**: Direct file access for configurations
+- **Model Cards**: Structured documentation parsing
+
+### Output Formats
+
+- **CycloneDX JSON**: Primary SBOM format
+- **Field Reports**: Human-readable analysis
+- **CSV Exports**: Batch processing results
+- **API Responses**: Structured JSON for integration
+
+This architecture provides a robust, configurable, and standards-compliant solution for AI model SBOM generation with comprehensive field extraction and scoring capabilities.
+