.github/instructions/copilot-instructions.md

Copilot Instructions for OpenSight-Deepfake-Detection-Models-Playground

Project Overview

This project is a modular, agent-driven toolkit for deepfake detection and digital forensics. It uses an ensemble of models, advanced forensic tools, and smart agents to provide explainable, extensible, and robust detection—optimized for integration with vision LLMs and multimodal AI agents.

Architecture & Key Components

• Entrypoint: app.py (Gradio app + MCP server).
• Forensics: forensics/ (e.g., ela.py, gradient.py, minmax.py, bitplane.py, wavelet.py). Each file implements a forensic technique, callable via LLM/MCP.
• Agents: agents/ (e.g., EnsembleMonitorAgent, ModelWeightManager, ContextualIntelligenceAgent, ForensicAnomalyDetectionAgent). Agents coordinate model weighting, context inference, and anomaly detection.
• Model Management: Models are registered in utils/registry.py and managed as an ensemble with dynamic, context-aware weighting.
• Utilities: utils/ (logging, augmentation, registry, health checks).

Data Flow & Prediction Pipeline

1. Image Preprocessing: Normalize to PIL RGB; optionally augment (rotate, noise, sharpen).
2. Agent Initialization: Monitoring, optimization, and context agents are set up.
3. Model Inference: Each model predicts independently; results tracked by agents.
4. Consensus: Model weights are dynamically adjusted based on context and agent feedback.
5. Forensic Analysis: Multiple forensic tools run in parallel; outputs analyzed for anomalies.
6. Logging: All results (images, predictions, agent data) are logged to Hugging Face datasets.

Developer Workflows

• Run the App:

  python app_optimized.py
  

• Dependencies: See requirements.txt (notably: gradio, PIL, numpy, torch, smolagents, etc.).
• Extending Forensics/Agents: Add new tools in forensics/, new agents in agents/, and register in the main app.
• Testing: Unit tests for agents/models are planned (see roadmap in README.md).

Project-Specific Patterns & Conventions

• Forensic Tool Naming: Use tool_* or descriptive names (e.g., tool_ela, tool_waveletnoise).
• Agent Classes: Use *Agent suffix (e.g., EnsembleMonitorAgent).
• API Exposure: Functions are exposed for LLM/MCP calls with clear parameter/return docs (see README.md).
• Consensus Logic: Model weights are contextually adjusted; see agent code for dynamic weighting.
• Logging: Inference data is logged to Hugging Face datasets for analysis.

Integration & Extension

• Add Models: Update ensemble logic and register in agent system.
• Add Forensic Tools: Implement in forensics/, expose via main app, and document parameters/returns.
• LLM/Multimodal Integration: Hybrid input strategies (e.g., ELA+RGB, metadata+image) are encouraged; see README.md for detailed tables and guidance.

References

• Forensic Techniques: See forensics/ for implementation details.
• Agent Logic: See agents/ for smart agent patterns.
• API & Usage: README.md documents all callable functions, parameters, and return types.
• Roadmap: Ongoing and planned features are tracked in README.md.

---

Tip: When extending or debugging, always check agent logic and consensus weighting, as these are central to system behavior.