Initial commit of multimodal multilingual PDF embedding pipeline

.gitignore

@@ -0,0 +1,28 @@
+# Python
+__pycache__/
+*.pyc
+*.pyo
+*.pyd
+.Python
+env/
+venv/
+*.egg
+*.egg-info/
+.env
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IDEs
+.idea/
+.vscode/
+
+# Output files
+output/
+embeddings_statistiques_multimodal.json
+extracted_graphs/
+extracted_tables/
+
+# API Keys (IMPORTANT!)
+*.env
+*.key

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Andre Sarr
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -1,5 +1,183 @@
----
-license: other
-license_name: aslic
-license_link: LICENSE
----
+# Multimodal & Multilingual PDF Embedding Pipeline
+
+This repository hosts a Python pipeline that extracts text, tables, and images from PDF documents, generates multimodal descriptions for visual content (tables and images), and then creates multilingual text embeddings for all extracted information. The generated embeddings are stored in a JSON file, ready for use in Retrieval Augmented Generation (RAG) systems or other downstream applications.
+
+**Key Features:**
+- **Multimodal:** Processes text, tables, and images from PDFs.
+- **Multilingual:** Leverages Google's `text-multilingual-embedding-002` model for embeddings, supporting a wide range of languages.
+- **Contextual Descriptions:** Uses Google Gemini (Gemini 1.5 Flash) to generate descriptive text for tables and images in French.
+- **Structured Output:** Stores embeddings and metadata (PDF source, page number, content type, links to extracted assets) in a comprehensive JSON format.
+
+## How it Works
+
+1.  **PDF Parsing:** Utilizes `PyMuPDF` to extract text blocks and images, and `Camelot` to accurately extract tabular data.
+2.  **Content Separation:** Distinguishes between plain text, tables, and non-table images.
+3.  **Multimodal Description (for Tables & Images):**
+    - For tables, the pipeline captures an image of the table and also uses its text representation.
+    - For standalone images (e.g., graphs, charts), it captures the image.
+    - These images are then sent to the `gemini-1.5-flash-latest` model (via `google.generativeai`) with specific prompts to generate rich, descriptive text in French.
+4.  **Multilingual Text Embedding:**
+    - The cleaned text content (original text chunks, or generated descriptions for tables/images) is then passed to the `text-multilingual-embedding-002` model (via Vertex AI).
+    - This model generates a high-dimensional embedding vector (768 dimensions) for each piece of content.
+5.  **JSON Output:** All generated embeddings, along with rich metadata (original PDF, page, content type, links to extracted assets), are compiled into a single JSON file.
+
+## Requirements & Setup
+
+This pipeline relies on **Google Cloud Platform** services and specific Python libraries. You will need:
+
+1.  **A Google Cloud Project:**
+    - Enable the **Vertex AI API**.
+    - Enable the **Generative Language API** (for Gemini 1.5 Flash descriptions).
+2.  **Authentication:**
+    - **Google Cloud Authentication:** The easiest way to run this in a Colab environment is using `google.colab.auth.authenticate_user()`. For local execution, ensure your Google Cloud SDK is configured and authenticated (`gcloud auth application-default login`).
+    - **Gemini API Key:** An API key for the Google AI Gemini models. You can get one from [Google AI Studio](https://aistudio.google.com/app/apikey). Set this as an environment variable or directly in the code (though environment variables are recommended for security).
+
+### Local Setup
+
+1.  **Clone the repository:**
+    ```bash
+    git clone [https://huggingface.co/](https://huggingface.co/)Anonymous1223334444/pdf-multimodal-multilingual-embedding-pipeline
+    cd pdf-multimodal-multilingual-embedding-pipeline
+    ```
+2.  **Install dependencies:**
+    ```bash
+    pip install -r requirements.txt
+    ```
+    **System-level dependencies for Camelot/PyMuPDF (Linux/Colab):**
+    You might need to install these system packages for `PyMuPDF` and `Camelot` to function correctly.
+    ```bash
+    # Update package list
+    sudo apt-get update
+    # Install Ghostscript (required by Camelot)
+    sudo apt-get install -y ghostscript
+    # Install python3-tk (required by some PyMuPDF functionalities)
+    sudo apt-get install -y python3-tk
+    # Install OpenCV (via apt, for camelot-py[cv])
+    sudo apt-get install -y libopencv-dev python3-opencv
+    ```
+    *Note: If you are running on Windows or macOS, the installation steps for `camelot-py` might differ. Refer to the [Camelot documentation](https://camelot-py.readthedocs.io/en/master/user/install-deps.html) for more details.*
+
+3.  **Set up Environment Variables:**
+    ```bash
+    export GOOGLE_CLOUD_PROJECT="your-gcp-project-id"
+    export VERTEX_AI_LOCATION="us-central1" # Or your preferred Vertex AI region (e.g., us-east4)
+    export GEMINI_API_KEY="your-gemini-api-key"
+    ```
+    Replace `your-gcp-project-id`, `us-central1`, and `your-gemini-api-key` with your actual values.
+
+4.  **Place your PDF files:**
+    Create a `docs` directory in the root of the repository and place your PDF documents inside it.
+    ```
+    pdf-multimodal-multilingual-embedding-pipeline/
+    ├── docs/
+    │   └── your_document.pdf
+    │   └── another_document.pdf
+    ```
+
+5.  **Run the pipeline:**
+    ```bash
+    python run_pipeline.py
+    ```
+    The generated embedding file (`embeddings_statistiques_multimodal.json`) and extracted assets will be saved in the `output/` directory.
+
+### Google Colab Usage
+
+A Colab notebook version of this pipeline is ideal for quick experimentation due to pre-configured environments.
+
+1.  **Open a new Google Colab notebook.**
+2.  **Install system dependencies:**
+    ```python
+    !pip uninstall -y camelot camelot-py # Ensure clean install
+    !pip install PyMuPDF
+    !apt-get update
+    !apt-get install -y ghostscript python3-tk libopencv-dev python3-opencv
+    !pip install camelot-py[cv] google-cloud-aiplatform tiktoken pandas beautifulsoup4 Pillow
+    ```
+3.  **Authenticate:**
+    ```python
+    from google.colab import auth
+    auth.authenticate_user()
+    ```
+4.  **Set your API Key and Project/Location:**
+    ```python
+    import os
+    # Replace with your actual Gemini API key
+    os.environ["GENAI_API_KEY"] = "YOUR_GEMINI_API_KEY_HERE"
+    # Replace with your actual Google Cloud Project ID
+    os.environ["GOOGLE_CLOUD_PROJECT"] = "YOUR_GCP_PROJECT_ID_HERE"
+    # Set your preferred Vertex AI location (e.g., "us-central1", "us-east4")
+    os.environ["VERTEX_AI_LOCATION"] = "us-central1"
+    ```
+5.  **Upload your PDF files:**
+    You can use the Colab file upload feature or mount Google Drive. Ensure your PDFs are in a directory named `docs` within `/content/`.
+    ```python
+    # Example for uploading
+    from google.colab import files
+    import os
+    PDF_DIRECTORY = Path("/content/docs")
+    PDF_DIRECTORY.mkdir(parents=True, exist_ok=True)
+    uploaded = files.upload()
+    for filename in uploaded.keys():
+        os.rename(filename, PDF_DIRECTORY / filename)
+    ```
+6.  **Copy and paste the code from `run_pipeline.py` (and `src/` files if you don't use modules) into Colab cells and execute.**
+
+## Output
+
+The pipeline will generate:
+- `embeddings_statistiques_multimodal.json`: A JSON file containing all generated embeddings and their metadata.
+- `output/extracted_graphs/`: Directory containing extracted images (PNG format).
+- `output/extracted_tables/`: Directory containing HTML representations of extracted tables.
+
+## Example `embeddings_statistiques_multimodal.json` Entry
+
+```json
+[
+  {
+    "pdf_file": "sample.pdf",
+    "page_number": 1,
+    "chunk_id": "text_0",
+    "content_type": "text",
+    "text_content": "This is a chunk of text extracted from the first page of the document...",
+    "embedding": [0.123, -0.456, ..., 0.789],
+    "pdf_title": "Sample Document",
+    "pdf_subject": "Data Analysis",
+    "pdf_keywords": "statistics, report"
+  },
+  {
+    "pdf_file": "sample.pdf",
+    "page_number": 2,
+    "chunk_id": "table_0",
+    "content_type": "table",
+    "text_content": "Description en français du tableau: Ce tableau présente les ventes mensuelles par région. Il inclut les colonnes Mois, Région, et Ventes. La région Nord a la plus forte croissance...",
+    "embedding": [-0.987, 0.654, ..., 0.321],
+    "table_html_url": "/static/extracted_tables/sample_p2_table0.html",
+    "image_url": "/static/extracted_graphs/sample_p2_table0.png",
+    "pdf_title": "Sample Document",
+    "pdf_subject": "Data Analysis",
+    "pdf_keywords": "statistics, report"
+  },
+  {
+    "pdf_file": "sample.pdf",
+    "page_number": 3,
+    "chunk_id": "image_0",
+    "content_type": "image",
+    "text_content": "Description en français de l'image: Ce graphique est un histogramme montrant la répartition des âges dans la population. L'axe des X représente les tranches d'âge et l'axe des Y la fréquence. La majorité de la population se situe entre 25 et 40 ans.",
+    "embedding": [0.456, -0.789, ..., 0.123],
+    "image_url": "/static/extracted_graphs/sample_p3_img0.png",
+    "pdf_title": "Sample Document",
+    "pdf_subject": "Data Analysis",
+    "pdf_keywords": "statistics, report"
+  }
+]
+```
+
+# Acknowledgments
+This pipeline leverages the power of:
+- Google Cloud Vertex AI
+- Google AI Gemini Models
+- PyMuPDF
+- Camelot
+- Tiktoken
+- Pandas
+- BeautifulSoup

requirements.txt

@@ -0,0 +1,8 @@
+PyMuPDF
+camelot-py[cv]
+google-cloud-aiplatform
+google-generativeai
+tiktoken
+pandas
+beautifulsoup4
+Pillow

run_pipeline.py

@@ -0,0 +1,222 @@
+import os
+import json
+import traceback
+from pathlib import Path
+import tiktoken
+
+# Import functions from your src directory
+from src.pdf_processor import extract_page_data_pymupdf, clean_text
+from src.embedding_utils import initialize_clients, token_chunking, generate_multimodal_description, generate_text_embedding, ENCODING_NAME, MAX_TOKENS_NORMAL
+
+# --- Configuration ---
+# You can set these directly or get them from environment variables (recommended)
+PROJECT_ID = os.getenv("GOOGLE_CLOUD_PROJECT")
+LOCATION = os.getenv("VERTEX_AI_LOCATION")
+GENAI_API_KEY = os.getenv("GENAI_API_KEY") # For Gemini API
+
+# Path configuration
+BASE_DIR = Path.cwd() # Current working directory of the script
+PDF_DIRECTORY = BASE_DIR / "docs"
+OUTPUT_DIR = BASE_DIR / "output" # New output directory for generated files
+EMBEDDINGS_FILE_PATH = OUTPUT_DIR / "embeddings_statistiques_multimodal.json"
+
+# Directory to save extracted images and tables HTML (within output)
+IMAGE_SAVE_SUBDIR = "extracted_graphs"
+TABLE_SAVE_SUBDIR = "extracted_tables"
+# Absolute paths for saving
+IMAGE_SAVE_DIR = OUTPUT_DIR / IMAGE_SAVE_SUBDIR
+TABLE_SAVE_DIR = OUTPUT_DIR / TABLE_SAVE_SUBDIR
+
+# --- Main Processing Function ---
+
+def process_pdfs_in_directory(directory):
+    """Main processing pipeline for all PDFs in a directory."""
+    all_embeddings_data = []
+    processed_files = 0
+    pdf_files = list(directory.glob("*.pdf"))
+    total_files = len(pdf_files)
+
+    if total_files == 0:
+        print(f"Aucun fichier PDF trouvé dans le répertoire : {directory}")
+        return []
+
+    for pdf_file_path in pdf_files:
+        processed_files += 1
+        print(f"\nTraitement de {pdf_file_path.name} ({processed_files}/{total_files})...")
+
+        page_data_list = extract_page_data_pymupdf(pdf_file_path, IMAGE_SAVE_DIR, TABLE_SAVE_DIR, IMAGE_SAVE_SUBDIR, TABLE_SAVE_SUBDIR)
+
+        if not page_data_list:
+            print(f"  Aucune donnée extraite de {pdf_file_path.name}.")
+            continue
+
+        for page_data in page_data_list:
+            pdf_file = page_data['pdf_file']
+            page_num = page_data['page_number']
+            page_text = page_data['text']
+            images = page_data['images'] # List of non-table image dicts
+            tables = page_data['tables'] # List of table dicts
+            pdf_title = page_data.get('pdf_title')
+            pdf_subject = page_data.get('pdf_subject')
+            pdf_keywords = page_data.get('pdf_keywords')
+
+            print(f"  Génération des descriptions et embeddings pour la page {page_num}...")
+
+            # Process tables: Generate description and then embedding
+            for table_idx, table in enumerate(tables):
+                table_image_bytes = table.get('image_bytes')
+                table_text_repr = table.get('table_text_representation', '')
+                table_html_url = table.get('table_html_url')
+
+                description = None
+                if table_image_bytes:
+                    prompt = "Décrivez en français le contenu et la structure de ce tableau. Mettez l'accent sur les données principales et les tendances si visibles."
+                    print(f"  Page {page_num}: Génération de la description multimodale pour le tableau {table_idx}...")
+                    description = generate_multimodal_description(table_image_bytes, prompt)
+                elif table_text_repr:
+                    prompt = f"Décrivez en français le contenu et la structure de ce tableau basé sur sa représentation textuelle:\n{table_text_repr[:1000]}..."
+                    print(f"  Page {page_num}: Génération de la description textuelle pour le tableau {table_idx} (fallback)...")
+                    # Use the multimodal model with text-only input (via google.generativeai)
+                    if GENAI_API_KEY:
+                        try:
+                            model = genai.GenerativeModel("models/gemini-1.5-flash-latest") # Explicitly use the model
+                            response = model.generate_content(prompt)
+                            description = response.text.strip()
+                        except Exception as e:
+                            print(f"  Erreur lors de la génération de description textuelle pour le tableau {table_idx}: {e}")
+                            description = None
+                    else:
+                        print("  Skipping text description generation for table: GEMINI_API_KEY is not set.")
+                        description = None
+
+
+                if description:
+                    print(f"  Page {page_num}: Description générée pour le tableau {table_idx}.")
+                    embedding_vector = generate_text_embedding(description) # max_retries, delay are defaults
+
+                    if embedding_vector is not None:
+                        chunk_data = {
+                            "pdf_file": pdf_file,
+                            "page_number": page_num,
+                            "chunk_id": f"table_{table_idx}",
+                            "content_type": "table",
+                            "text_content": description,
+                            "embedding": embedding_vector,
+                            "table_html_url": table_html_url,
+                            "image_url": table.get('image_url'),
+                            "pdf_title": pdf_title,
+                            "pdf_subject": pdf_subject,
+                            "pdf_keywords": pdf_keywords
+                        }
+                        all_embeddings_data.append(chunk_data)
+                        print(f"  Page {page_num}: Embedding généré pour la description du tableau {table_idx}.")
+                    else:
+                        print(f"  Page {page_num}: Échec de la génération de l'embedding pour la description du tableau {table_idx}. Chunk ignoré.")
+                else:
+                    print(f"  Page {page_num}: Aucune description générée pour le tableau {table_idx}. Chunk ignoré.")
+
+
+            # Process images (non-table): Generate description and then embedding
+            for img_idx, image in enumerate(images):
+                image_bytes = image.get('image_bytes')
+                image_url = image.get('image_url')
+
+                if image_bytes:
+                    prompt = "Décrivez en français le contenu de cette image. S'il s'agit d'un graphique, décrivez le type de graphique (histogramme, courbe, etc.), les axes, les légendes et les principales informations ou tendances visibles."
+                    print(f"  Page {page_num}: Génération de la description multimodale pour l'image {img_idx}...")
+                    description = generate_multimodal_description(image_bytes, prompt)
+
+                    if description:
+                        print(f"  Page {page_num}: Description générée pour l'image {img_idx}.")
+                        embedding_vector = generate_text_embedding(description) # max_retries, delay are defaults
+
+                        if embedding_vector is not None:
+                            chunk_data = {
+                                "pdf_file": pdf_file,
+                                "page_number": page_num,
+                                "chunk_id": f"image_{img_idx}",
+                                "content_type": "image",
+                                "text_content": description,
+                                "embedding": embedding_vector,
+                                "image_url": image_url,
+                                "pdf_title": pdf_title,
+                                "pdf_subject": pdf_subject,
+                                "pdf_keywords": pdf_keywords
+                            }
+                            all_embeddings_data.append(chunk_data)
+                            print(f"  Page {page_num}: Embedding généré pour la description de l'image {img_idx}.")
+                        else:
+                            print(f"  Page {page_num}: Échec de la génération de l'embedding pour la description de l'image {img_idx}. Chunk ignoré.")
+                    else:
+                        print(f"  Page {page_num}: Aucune description générée pour l'image {img_idx}. Chunk ignoré.")
+
+
+            # Process regular text: Chunk and then generate embeddings
+            if page_text:
+                try:
+                    encoding = tiktoken.get_encoding(ENCODING_NAME)
+                    text_chunks = token_chunking(page_text, MAX_TOKENS_NORMAL, encoding)
+                except Exception as e:
+                    print(f"Erreur lors du chunking du texte de la page {page_num} : {e}. Utilisation du chunking simple.")
+                    text_chunks = [page_text]
+
+
+                for chunk_idx, chunk_content in enumerate(text_chunks):
+                    print(f"  Page {page_num}: Génération de l'embedding pour le chunk de texte {chunk_idx}...")
+                    embedding_vector = generate_text_embedding(chunk_content) # max_retries, delay are defaults
+
+                    if embedding_vector is not None:
+                        chunk_data = {
+                            "pdf_file": pdf_file,
+                            "page_number": page_num,
+                            "chunk_id": f"text_{chunk_idx}",
+                            "content_type": "text",
+                            "text_content": chunk_content,
+                            "embedding": embedding_vector,
+                            "pdf_title": pdf_title,
+                            "pdf_subject": pdf_subject,
+                            "pdf_keywords": pdf_keywords
+                        }
+                        all_embeddings_data.append(chunk_data)
+                        print(f"  Page {page_num}: Chunk de texte {chunk_idx} traité avec succès.")
+                    else:
+                        print(f"  Page {page_num}: Échec de la génération de l'embedding pour le chunk de texte {chunk_idx}. Chunk ignoré.")
+
+
+            print(f"  Page {page_num} terminée. Éléments traités : {len(tables)} tableaux, {len(images)} images, {len(text_chunks)} chunks de texte.")
+
+
+    return all_embeddings_data
+
+# --- Main Execution ---
+if __name__ == "__main__":
+    print("Démarrage du traitement PDF multimodal avec génération de descriptions et embeddings textuels multilingues...")
+
+    # Validate and create directories
+    if not PDF_DIRECTORY.is_dir():
+        print(f"❌ ERREUR: Répertoire PDF non trouvé ou n'est pas un répertoire : {PDF_DIRECTORY}")
+        exit(1)
+
+    OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
+    IMAGE_SAVE_DIR.mkdir(parents=True, exist_ok=True)
+    TABLE_SAVE_DIR.mkdir(parents=True, exist_ok=True)
+    print(f"Répertoire de sortie : {OUTPUT_DIR}")
+    print(f"Répertoire de sauvegarde des images : {IMAGE_SAVE_DIR}")
+    print(f"Répertoire de sauvegarde des tableaux (HTML) : {TABLE_SAVE_DIR}")
+
+    # Initialize clients for Vertex AI and GenAI
+    initialize_clients(PROJECT_ID, LOCATION, GENAI_API_KEY)
+
+    final_embeddings = process_pdfs_in_directory(PDF_DIRECTORY)
+
+    if final_embeddings:
+        print(f"\nTotal d'embeddings générés : {len(final_embeddings)}.")
+        try:
+            with EMBEDDINGS_FILE_PATH.open('w', encoding='utf-8') as f:
+                json.dump(final_embeddings, f, indent=2, ensure_ascii=False)
+            print(f"Embeddings sauvegardés avec succès dans : {EMBEDDINGS_FILE_PATH}")
+        except Exception as e:
+            print(f"\nErreur lors de la sauvegarde du fichier JSON d'embeddings : {e}")
+            traceback.print_exc()
+    else:
+        print("\nAucun embedding n'a été généré.")

src/embedding_utils.py

@@ -0,0 +1,205 @@
+import os
+import time
+import random
+import traceback
+import tiktoken
+
+import google.generativeai as genai
+import vertexai
+from vertexai.language_models import TextEmbeddingModel
+
+# Configuration (will be initialized from run_pipeline.py)
+# For module, these should ideally be arguments or imported from a config
+# GENAI_API_KEY = os.getenv("GENAI_API_KEY")
+# PROJECT_ID = os.getenv("GOOGLE_CLOUD_PROJECT")
+# LOCATION = os.getenv("VERTEX_AI_LOCATION")
+
+MULTIMODAL_MODEL_GENAI = "models/gemini-1.5-flash-latest"
+TEXT_EMBEDDING_MODEL_VERTEXAI = "text-multilingual-embedding-002"
+EMBEDDING_DIMENSION = 768 # text-multilingual-embedding-002 has 768 dimensions
+
+MAX_TOKENS_NORMAL = 500
+ENCODING_NAME = "cl100k_base"
+
+# Global client for Vertex AI Text Embedding Model
+text_embedding_model_client = None
+
+def initialize_clients(project_id, location, genai_api_key):
+    """Initializes Vertex AI and GenAI clients."""
+    global text_embedding_model_client
+    
+    if genai_api_key:
+        genai.configure(api_key=genai_api_key)
+        print("✓ Google Generative AI configured.")
+    else:
+        print("⚠️ AVERTISSEMENT: La clé API Gemini n'est pas définie. La génération de descriptions multimodales échouera.")
+
+    if project_id and location:
+        try:
+            vertexai.init(project=project_id, location=location)
+            print(f"✓ Vertex AI SDK initialisé pour le projet {project_id} dans la région {location}.")
+            text_embedding_model_client = TextEmbeddingModel.from_pretrained(TEXT_EMBEDDING_MODEL_VERTEXAI)
+            print(f"✓ Modèle d'embedding textuel Vertex AI '{TEXT_EMBEDDING_MODEL_VERTEXAI}' chargé avec succès.")
+        except Exception as e:
+            print(f"❌ ERREUR: Échec de l'initialisation du Vertex AI SDK ou du chargement du modèle d'embedding textuel : {str(e)}")
+            print("⚠️ La génération d'embeddings textuels échouera.")
+            text_embedding_model_client = None
+    else:
+        print("⚠️ Vertex AI SDK non initialisé car l'ID du projet Google Cloud ou la localisation sont manquants.")
+        print("⚠️ La génération d'embeddings textuels échouera.")
+        text_embedding_model_client = None
+
+
+def token_chunking(text, max_tokens, encoding):
+    """Chunk text based on token count with smarter boundaries (sentences, paragraphs)"""
+    if not text:
+        return []
+
+    tokens = encoding.encode(text)
+    chunks = []
+    start_token_idx = 0
+
+    while start_token_idx < len(tokens):
+        end_token_idx = min(start_token_idx + max_tokens, len(tokens))
+
+        if end_token_idx < len(tokens):
+            look_ahead_limit = min(start_token_idx + max_tokens * 2, len(tokens))
+            text_segment_to_check = encoding.decode(tokens[start_token_idx:look_ahead_limit])
+
+            paragraph_break = text_segment_to_check.rfind('\n\n', 0, len(text_segment_to_check) - (look_ahead_limit - (start_token_idx + max_tokens)))
+            if paragraph_break != -1:
+                tokens_up_to_break = encoding.encode(text_segment_to_check[:paragraph_break])
+                end_token_idx = start_token_idx + len(tokens_up_to_break)
+            else:
+                sentence_end = re.search(r'[.!?]\s+', text_segment_to_check[:len(text_segment_to_check) - (look_ahead_limit - (start_token_idx + max_tokens))][::-1])
+                if sentence_end:
+                    char_index_in_segment = len(text_segment_to_check) - 1 - sentence_end.start()
+                    tokens_up_to_end = encoding.encode(text_segment_to_check[:char_index_in_segment + 1])
+                    end_token_idx = start_token_idx + len(tokens_up_to_end)
+
+        current_chunk_tokens = tokens[start_token_idx:end_token_idx]
+        chunk_text = encoding.decode(current_chunk_tokens).strip()
+
+        if chunk_text:
+            chunks.append(chunk_text)
+
+        if start_token_idx == end_token_idx:
+            start_token_idx += 1
+        else:
+            start_token_idx = end_token_idx
+
+    return chunks
+
+
+def generate_multimodal_description(image_bytes, prompt_text, multimodal_model_genai_name=MULTIMODAL_MODEL_GENAI, max_retries=5, delay=10):
+    """
+    Generate a text description for an image using a multimodal model (google.generativeai).
+    Returns description text or None if all retries fail or API key is missing.
+    """
+    if not genai.api_key: # Check if API key is configured
+        print("  Skipping multimodal description generation: GEMINI_API_KEY is not set.")
+        return None
+
+    for attempt in range(max_retries):
+        try:
+            time.sleep(delay + random.uniform(0, 5))
+
+            content = [
+                prompt_text,
+                {
+                    'mime_type': 'image/png',
+                    'data': image_bytes
+                }
+            ]
+
+            model = genai.GenerativeModel(multimodal_model_genai_name)
+            response = model.generate_content(content)
+
+            description = response.text.strip()
+
+            if description:
+                return description
+            else:
+                print(f"  Tentative {attempt+1}/{max_retries}: Réponse vide ou inattendue du modèle multimodal.")
+                if attempt < max_retries - 1:
+                    retry_delay = delay * (2 ** attempt) + random.uniform(1, 5)
+                    print(f"  Réessai dans {retry_delay:.2f}s...")
+                    time.sleep(retry_delay)
+                continue
+                # else:
+                #    print(f"  Toutes les {max_retries} tentatives ont échoué pour générer la description.")
+                #    return None
+
+
+        except Exception as e:
+            error_msg = str(e)
+            print(f"  Tentative {attempt+1}/{max_retries} échouée pour la description : {error_msg}")
+
+            if "429" in error_msg or "quota" in error_msg.lower() or "rate limit" in error_msg.lower() or "unavailable" in error_msg.lower() or "internal error" in error_msg.lower():
+                if attempt < max_retries - 1:
+                    retry_delay = delay * (2 ** attempt) + random.uniform(1, 5)
+                    print(f"  Erreur d'API retryable détectée. Réessai dans {retry_delay:.2f}s...")
+                    time.sleep(retry_delay)
+                    continue
+                # else:
+                #     print(f"  Toutes les {max_retries} tentatives ont échoué pour la description.")
+                #     return None
+
+            else:
+                print(f"  Erreur d'API non retryable détectée : {error_msg}")
+                traceback.print_exc()
+                return None
+
+    print(f"  Toutes les {max_retries} tentatives ont échoué pour la description (fin de boucle).")
+    return None
+
+
+def generate_text_embedding(text_content, max_retries=5, delay=5):
+    """
+    Generate text embedding using the Vertex AI multilingual embedding model.
+    Returns embedding vector (list) or None if all retries fail or client is not initialized.
+    """
+    global text_embedding_model_client # Ensure we are using the global client
+
+    if not text_embedding_model_client:
+        print("  Skipping text embedding generation: Vertex AI embedding client is not initialized.")
+        return None
+
+    if not text_content or not text_content.strip():
+        return None # Cannot embed empty text
+
+    for attempt in range(max_retries):
+        try:
+            time.sleep(delay + random.uniform(0, 2))
+
+            embeddings = text_embedding_model_client.get_embeddings( # Corrected method name
+                [text_content] # Removed task_type
+            )
+
+            if embeddings and len(embeddings) > 0 and hasattr(embeddings[0], 'values') and isinstance(embeddings[0].values, list) and len(embeddings[0].values) == EMBEDDING_DIMENSION:
+                return embeddings[0].values
+            else:
+                print(f"  Tentative {attempt+1}/{max_retries}: Format d'embedding Vertex AI inattendu. Réponse : {embeddings}")
+                return None
+
+        except Exception as e:
+            error_msg = str(e)
+            print(f"  Tentative {attempt+1}/{max_retries} échouée pour l'embedding Vertex AI : {error_msg}")
+
+            if "429" in error_msg or "quota" in error_msg.lower() or "rate limit" in error_msg.lower() or "unavailable" in error_msg.lower() or "internal error" in error_msg.lower():
+                if attempt < max_retries - 1:
+                    retry_delay = delay * (2 ** attempt) + random.uniform(1, 5)
+                    print(f"  Erreur d'API Vertex AI retryable détectée. Réessai dans {retry_delay:.2f}s...")
+                    time.sleep(retry_delay)
+                    continue
+                # else:
+                #     print(f"  Toutes les {max_retries} tentatives ont échoué pour l'embedding Vertex AI.")
+                #     return None
+
+            else:
+                print(f"  Erreur d'API Vertex AI non retryable détectée : {error_msg}")
+                traceback.print_exc()
+                return None
+
+    print(f"  Toutes les {max_retries} tentatives ont échoué pour l'embedding Vertex AI (fin de boucle).")
+    return None

src/pdf_processor.py

@@ -0,0 +1,261 @@
+import fitz  # PyMuPDF
+import camelot  # For table extraction
+import pandas as pd
+from bs4 import BeautifulSoup
+import re
+from pathlib import Path
+import traceback
+
+# Path configuration (assuming these are passed or relative to run_pipeline.py)
+# For module, these should ideally be arguments or imported from a config
+# BASE_DIR = Path("/content/")
+# PDF_DIRECTORY = BASE_DIR / "docs"
+# IMAGE_SAVE_SUBDIR = "extracted_graphs"
+# TABLE_SAVE_SUBDIR = "extracted_tables"
+# STATIC_DIR = BASE_DIR / "static"
+# IMAGE_SAVE_DIR = STATIC_DIR / IMAGE_SAVE_SUBDIR
+# TABLE_SAVE_DIR = STATIC_DIR / TABLE_SAVE_SUBDIR
+
+# These should be passed as arguments or configured at a higher level
+IMAGE_MIN_WIDTH = 100   # Ignore very small images (likely logos/icons)
+IMAGE_MIN_HEIGHT = 100
+
+def clean_text(text):
+    """Normalize whitespace and clean text while preserving paragraph breaks"""
+    if not text:
+        return ""
+    # Replace tabs with spaces, but preserve paragraph breaks
+    text = text.replace('\t', ' ')
+    # Normalize multiple spaces to single spaces
+    text = re.sub(r' +', ' ', text)
+    # Preserve paragraph breaks but normalize them
+    text = re.sub(r'\n{3,}', '\n\n', text)
+    return text.strip()
+
+def extract_page_data_pymupdf(pdf_path, image_save_dir, table_save_dir, image_save_subdir, table_save_subdir):
+    """Extract text, tables and save images from each page using PyMuPDF and Camelot."""
+    page_data_list = []
+    try:
+        doc = fitz.open(pdf_path)
+        metadata = doc.metadata or {}
+        pdf_data = {
+            'pdf_title': metadata.get('title', pdf_path.name),
+            'pdf_subject': metadata.get('subject', 'Statistiques'),
+            'pdf_keywords': metadata.get('keywords', '')
+        }
+
+        for page_num in range(len(doc)):
+            page = doc.load_page(page_num)
+            page_index = page_num + 1  # 1-based index
+
+            print(f"  Extraction des données de la page {page_index}...")
+
+            # Extract tables first
+            table_data = extract_tables_and_images_from_page(pdf_path, page, page_index, table_save_dir, image_save_dir, image_save_subdir, table_save_subdir)
+
+            # Track table regions to avoid double-processing text
+            table_regions = []
+            for item in table_data:
+                if 'rect' in item and item['rect'] and len(item['rect']) == 4:
+                    table_regions.append(fitz.Rect(item['rect']))
+                else:
+                    print(f"  Warning: Invalid rect for table on page {page_index}")
+
+
+            # Extract text excluding table regions
+            page_text = ""
+            if table_regions:
+                # Get text blocks
+                blocks = page.get_text("blocks")
+                for block in blocks:
+                    block_rect = fitz.Rect(block[:4])
+                    is_in_table = False
+                    for table_rect in table_regions:
+                        if block_rect.intersects(table_rect):
+                            is_in_table = True
+                            break
+                    if not is_in_table:
+                        page_text += block[4] + "\n" # Add text content
+            else:
+                # If no tables, get all text
+                page_text = page.get_text("text")
+
+            page_text = clean_text(page_text)
+
+
+            # Extract and save images (excluding those identified as tables)
+            image_data = extract_images_from_page(pdf_path, page, page_index, image_save_dir, image_save_subdir, excluded_rects=table_regions)
+
+
+            page_data_list.append({
+                'pdf_file': pdf_path.name,
+                'page_number': page_index,
+                'text': page_text,
+                'images': image_data, # Includes non-table images
+                'tables': [item for item in table_data if item['content_type'] == 'table'], # Only table data here
+                'pdf_title': pdf_data.get('pdf_title'),
+                'pdf_subject': pdf_data.get('pdf_subject'),
+                'pdf_keywords': pdf_data.get('pdf_keywords')
+            })
+        doc.close()
+    except Exception as e:
+        print(f"Erreur lors du traitement du PDF {pdf_path.name} avec PyMuPDF : {str(e)}")
+        traceback.print_exc() # Print traceback for debugging
+    return page_data_list
+
+
+def extract_tables_and_images_from_page(pdf_path, page, page_num, table_save_dir, image_save_dir, image_save_subdir, table_save_subdir):
+    """Extract tables using Camelot and capture images of table areas."""
+    table_and_image_data = []
+    try:
+        tables = camelot.read_pdf(
+            str(pdf_path),
+            pages=str(page_num),
+            flavor='lattice',
+        )
+
+        if len(tables) == 0:
+            tables = camelot.read_pdf(
+                str(pdf_path),
+                pages=str(page_num),
+                flavor='stream'
+            )
+
+        for i, table in enumerate(tables):
+            if table.accuracy < 70:
+                print(f"  Skipping low accuracy table ({table.accuracy:.2f}%) on page {page_num}")
+                continue
+
+            table_bbox = table.parsing_report.get('page_bbox', [0, 0, 0, 0])
+            if not table_bbox or len(table_bbox) != 4:
+                print(f"  Warning: Invalid bounding box for table {i} on page {page_num}. Skipping image capture.")
+                table_rect = None
+            else:
+                table_rect = fitz.Rect(table_bbox)
+
+            safe_pdf_name = "".join(c if c.isalnum() else "_" for c in pdf_path.stem)
+            table_html_filename = f"{safe_pdf_name}_p{page_num}_table{i}.html"
+            table_html_save_path = table_save_dir / table_html_filename
+            relative_html_url_path = f"/static/{table_save_subdir}/{table_html_filename}"
+
+            table_image_filename = f"{safe_pdf_name}_p{page_num}_table{i}.png"
+            table_image_save_path = image_save_dir / table_image_filename
+            relative_image_url_path = f"/static/{image_save_subdir}/{table_image_filename}"
+
+
+            df = table.df
+            html = f"<caption>Table extrait de {pdf_path.name}, page {page_num}</caption>\n" + df.to_html(index=False)
+            soup = BeautifulSoup(html, 'html.parser')
+            table_tag = soup.find('table')
+            if table_tag:
+                table_tag['class'] = 'table table-bordered table-striped'
+                table_tag['style'] = 'width:100%; border-collapse:collapse;'
+
+                style_tag = soup.new_tag('style')
+                style_tag.string = """
+                .table { border-collapse: collapse; width: 100%; margin-bottom: 1rem;}
+                .table caption { caption-side: top; padding: 0.5rem; text-align: left; font-weight: bold; }
+                .table th, .table td { border: 1px solid #ddd; padding: 8px; text-align: left; }
+                .table th { background-color: #f2f2f2; font-weight: bold; }
+                .table-striped tbody tr:nth-of-type(odd) { background-color: rgba(0,0,0,.05); }
+                .table-responsive { overflow-x: auto; margin-bottom: 1rem; }
+                """
+                soup.insert(0, style_tag)
+
+                div = soup.new_tag('div')
+                div['class'] = 'table-responsive'
+                table_tag.wrap(div)
+
+                with open(table_html_save_path, 'w', encoding='utf-8') as f:
+                    f.write(str(soup))
+            else:
+                print(f"  Warning: Could not find table tag in HTML for table on page {page_num}. Skipping HTML save.")
+                continue
+
+            table_image_bytes = None
+            if table_rect:
+                try:
+                    pix = page.get_pixmap(clip=table_rect)
+                    table_image_bytes = pix.tobytes(format='png')
+
+                    with open(table_image_save_path, "wb") as img_file:
+                        img_file.write(table_image_bytes)
+
+                except Exception as img_capture_e:
+                    print(f"  Erreur lors de la capture d'image du tableau {i} page {page_num} : {img_capture_e}")
+                    traceback.print_exc()
+                    table_image_bytes = None
+
+
+            table_and_image_data.append({
+                'content_type': 'table',
+                'table_html_url': relative_html_url_path,
+                'table_text_representation': df.to_string(index=False),
+                'rect': [table_rect.x0, table_rect.y0, table_rect.x1, table_rect.y1] if table_rect else None,
+                'accuracy': table.accuracy,
+                'image_bytes': table_image_bytes,
+                'image_url': relative_image_url_path if table_image_bytes else None
+            })
+
+        return table_and_image_data
+
+    except Exception as e:
+        print(f"  Erreur lors de l'extraction des tableaux de la page {page_num} : {str(e)}")
+        traceback.print_exc()
+        return []
+
+def extract_images_from_page(pdf_path, page, page_num, image_save_dir, image_save_subdir, excluded_rects=[]):
+    """Extract and save images from a page, excluding specified regions (like tables)."""
+    image_data = []
+    image_list = page.get_images(full=True)
+
+    for img_index, img_info in enumerate(image_list):
+        xref = img_info[0]
+        try:
+            base_image = page.parent.extract_image(xref)
+            image_bytes = base_image["image"]
+            image_ext = base_image["ext"]
+            width = base_image["width"]
+            height = base_image["height"]
+
+            if width < IMAGE_MIN_WIDTH or height < IMAGE_MIN_HEIGHT:
+                continue
+
+            img_rect = None
+            img_rects = page.get_image_rects(xref)
+            if img_rects:
+                img_rect = img_rects[0]
+
+            if img_rect is None:
+                print(f"  Warning: Could not find rectangle for image {img_index} on page {page_num}. Skipping.")
+                continue
+
+            is_excluded = False
+            for excluded_rect in excluded_rects:
+                if img_rect.intersects(excluded_rect):
+                    is_excluded = True
+                    break
+            if is_excluded:
+                print(f"  Image {img_index} on page {page_num} is within an excluded region (e.g., table). Skipping.")
+                continue
+
+            safe_pdf_name = "".join(c if c.isalnum() else "_" for c in pdf_path.stem)
+            image_filename = f"{safe_pdf_name}_p{page_num}_img{img_index}.{image_ext}"
+            image_save_path = image_save_dir / image_filename
+            relative_url_path = f"/static/{image_save_subdir}/{image_filename}"
+
+            with open(image_save_path, "wb") as img_file:
+                img_file.write(image_bytes)
+
+            image_data.append({
+                'content_type': 'image',
+                'image_url': relative_url_path,
+                'rect': [img_rect.x0, img_rect.y0, img_rect.x1, img_rect.y1],
+                'image_bytes': image_bytes
+            })
+
+        except Exception as img_save_e:
+            print(f"  Erreur lors du traitement de l'image {img_index} de la page {page_num} : {img_save_e}")
+            traceback.print_exc()
+
+    return image_data