Upload folder using huggingface_hub

.cursorignore

@@ -0,0 +1,6 @@
+screenplays/scripts/*
+screenplays/scripts/twelve_monkeys.txt
+screenplays/scripts/thefifthelement.txt
+screenplays/scripts/AmericanBeauty_final.txt
+screenplays/scripts/
+

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+ragtag4 filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -0,0 +1,2 @@
+*.pdf
+.env

Dockerfile

@@ -0,0 +1,17 @@
+# golang image and setup for the gin server
+FROM golang:1.24.1
+
+# Set the working directory in the container
+WORKDIR /app
+
+# Copy the current directory contents into the container at /app
+COPY . /app
+
+RUN go mod download
+
+RUN go build -o ragtag
+
+EXPOSE 8080
+
+CMD ["./ragtag"]
+

Dockerfile.ollama

@@ -0,0 +1,10 @@
+FROM ollama/ollama:latest
+
+# Install curl for healthcheck
+RUN apt-get update && apt-get install -y curl
+
+# Copy the entrypoint script
+COPY ollama-entrypoint.sh /ollama-entrypoint.sh
+RUN chmod +x /ollama-entrypoint.sh
+
+ENTRYPOINT ["/ollama-entrypoint.sh"]

README.md

@@ -1,10 +1,123 @@
----
-title: Ragtag4
-emoji: 👁
-colorFrom: indigo
-colorTo: gray
-sdk: docker
-pinned: false
----
-
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# what
+
+RAG is easy! Run ollama llama3.1 in golang with a postgres database.
+
+This is a simple example of how to use ollama with a postgres database to create a RAG system. It should be considered a starting point and not a full-featured system. It can be used and adapted for any data related use case for using llm's to answer questions about data.
+
+- Cool feature:  
+You can use the title as a category filter and if you add an '@' to the query, it will filter vector docs by that matched title, so for example: 
+`Is @antibionic a good company?` and it sets only docs and vectors with antibionic in the title field as the ones for the entire chat session moving forward. You can see this in the screenshots below. This means you can have different types of documents and not be forced to chat with all of them at once.
+
+It is setup on docker compose and is ready to go if you skip to that section below.
+
+## gui
+
+The gui includes a document manager to add and remove documents from a database and a chat interface to interact with the system. It is in the format of a single page application and is built with html, css, and javascript. The style is in the format of an emulated terminal with a black background and white / green text.
+
+![Screenshot 2024-08-18 at 5 04 10 PM](https://github.com/user-attachments/assets/ea0b8b04-2dba-4e5c-88fd-037fe296be87)
+
+![PNG image](https://github.com/user-attachments/assets/da8a5c78-7365-459d-9f69-76956dc276df)
+
+## key files
+
+- [index.html](index.html) - a simple html gui to interact with the system
+- [main.go](main.go) - the main go file to interact with the system
+
+## what I did & learned
+
+- created a table with a vector column and learned it had to be 4096 size to match llama3.1
+- created a function to generate an embedding for a given text including splitting at 4096-bit chunks
+- created a function to query the table with an embedding and return the most similar texts within the chat
+- created a script to download screenplays
+- created a script to send screenplays to the database and auto-embed against with the llm
+- created web interface and document manager interface to do CRUD on docs and vectors and stream tokens back from chat prompt
+
+## tools
+
+Used https://cursor.sh and claude sonnet to help with codebase.
+
+## curl add embedding example with title and doc
+
+```bash
+curl -X POST http://localhost:8080/add_document \
+     -H "Content-Type: application/json" \
+     -d '{"title": "Screenplay Title", "doc_text": "INT. COFFEE SHOP - DAY\n\nJANE, 30s, sits at a corner table, typing furiously on her laptop. The cafe buzzes with quiet conversation.\n\nJOHN, 40s, enters, scanning the room. He spots Jane and approaches.\n\nJOHN\nMind if I join you?\n\nJane looks up, startled."}'
+```
+
+## curl upload document example
+
+```bash
+curl -X POST http://localhost:8080/upload_document \
+  -H "Content-Type: multipart/form-data" \
+  -F "title=Example Document" \
+  -F "file=@/path/to/your/document.pdf"
+```
+
+## curl query example
+
+```bash
+curl -X POST http://localhost:8080/query \
+     -H "Content-Type: application/json" \
+     -d '{"query": "What are the main characters in the screenplays that are in the coffeeshop?"}'
+```
+
+## filter query example by title
+
+```bash
+curl -X POST http://localhost:8080/query \
+  -H "Content-Type: application/json" \
+  -d '{
+    "query": "@screenplay Tell me about the main characters",
+    "sessionId": "1234567890"
+  }'
+```
+
+## sql table creation
+
+```sql
+CREATE DATABASE IF NOT EXISTS ragtag
+CREATE EXTENSION IF NOT EXISTS vector;
+CREATE TABLE IF NOT EXISTS items (
+    id SERIAL PRIMARY KEY,
+    title TEXT,
+    doc TEXT,
+    embedding vector(4096)
+);
+```
+
+## docker
+
+Do this first:  
+`docker build -t ragtag .`
+
+## docker compose
+
+`docker-compose up --build`  
+This will pull the llama3.1 model and start the ollama server. It will also start the go server and the gui and connect to postgres and tie it all together.
+
+## ports and versions
+
+ollama - 11434  
+go server - 8080  
+postgres - 5432  
+
+go version - 1.23.0  
+ollama version - 1.10.0  
+postgres version - 16.1 as the pgvector:pg16 docker image
+
+This has not been tested on other versions but should work on other versions of the software if you know what you are doing.
+
+## curl test no stream
+
+```bash
+curl http://localhost:11434/api/generate -d '{
+  "model": "llama3.1",
+  "prompt":"Why is the sky blue?",
+  "stream": false
+}'
+```
+
+### Helpers
+
+- [downloader.py](screenplays/downloader.py) - downloads a screenplay from a given URL
+- [send_screenplay.py](screenplays/send_screenplay.py) - sends a screenplay to the database

create-db.sql

@@ -0,0 +1,2 @@
+SELECT 'CREATE DATABASE ragtag'
+WHERE NOT EXISTS (SELECT FROM pg_database WHERE datname = 'ragtag')\gexec

describer.html

@@ -0,0 +1,146 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>Image Describer</title>
+    <style>
+        body {
+            font-family: 'Courier New', monospace;
+            background-color: #000;
+            color: #00ff00;
+            margin: 0;
+            padding: 20px;
+            display: flex;
+            flex-direction: column;
+            align-items: center;
+        }
+        h1 {
+            color: #00ff00;
+        }
+        #imageContainer {
+            width: 500px;
+            height: 500px;
+            border: 1px solid #00ff00;
+            display: flex;
+            justify-content: center;
+            align-items: center;
+            margin-bottom: 20px;
+        }
+        #uploadedImage {
+            max-width: 100%;
+            max-height: 100%;
+        }
+        #descriptionContainer {
+            width: 500px;
+            min-height: 100px;
+            border: 1px solid #00ff00;
+            padding: 10px;
+            margin-bottom: 20px;
+        }
+        #uploadForm {
+            margin-bottom: 20px;
+        }
+        input[type="file"] {
+            display: none;
+        }
+        label, button {
+            background-color: #003300;
+            color: #00ff00;
+            border: 1px solid #00ff00;
+            padding: 5px 10px;
+            cursor: pointer;
+        }
+        label:hover, button:hover {
+            background-color: #004400;
+        }
+        a {
+            color: #00ff00;
+            text-decoration: none;
+            margin-bottom: 20px;
+        }
+        a:hover {
+            text-decoration: underline;
+        }
+    </style>
+</head>
+<body>
+    <h1>Image Describer</h1>
+    <a href="/">Back to Chat</a>
+    <div id="imageContainer">
+        <img id="uploadedImage" src="" alt="Uploaded image will appear here">
+    </div>
+    <form id="uploadForm" enctype="multipart/form-data">
+        <label for="imageFile">Choose Image</label>
+        <input type="file" id="imageFile" name="file" accept="image/*" required>
+        <button type="submit">Describe Image</button>
+    </form>
+    <div id="descriptionContainer">
+        Description will appear here...
+    </div>
+
+    <script>
+        const uploadForm = document.getElementById('uploadForm');
+        const imageFile = document.getElementById('imageFile');
+        const uploadedImage = document.getElementById('uploadedImage');
+        const descriptionContainer = document.getElementById('descriptionContainer');
+
+        uploadForm.addEventListener('submit', async (e) => {
+            e.preventDefault();
+            const formData = new FormData(uploadForm);
+
+            try {
+                const response = await fetch('/describe_image', {
+                    method: 'POST',
+                    body: formData
+                });
+
+                if (!response.ok) {
+                    throw new Error('Network response was not ok');
+                }
+
+                const reader = response.body.getReader();
+                descriptionContainer.textContent = '';
+
+                let buffer = '';
+                while (true) {
+                    const { value, done } = await reader.read();
+                    if (done) break;
+
+                    buffer += new TextDecoder().decode(value);
+                    const lines = buffer.split('\n');
+                    buffer = lines.pop() || '';
+
+                    for (const line of lines) {
+                        if (line.startsWith('data:')) {
+                            const content = line.slice(5);
+                            if (content) {
+                                appendToDescription(content);
+                            }
+                        }
+                    }
+                }
+            } catch (error) {
+                console.error('Error:', error);
+                descriptionContainer.textContent = 'Error describing image';
+            }
+        });
+
+        function appendToDescription(content) {
+            descriptionContainer.textContent += content + '';
+            descriptionContainer.scrollTop = descriptionContainer.scrollHeight;
+        }
+
+        imageFile.addEventListener('change', (e) => {
+            const file = e.target.files[0];
+            if (file) {
+                const reader = new FileReader();
+                reader.onload = (e) => {
+                    uploadedImage.src = e.target.result;
+                };
+                reader.readAsDataURL(file);
+            }
+        });
+    </script>
+</body>
+</html>

docker-backup/Dockerfile

@@ -0,0 +1,18 @@
+# golang image and setup for the gin server
+FROM golang:1.23.0
+
+# Set the working directory in the container
+WORKDIR /app
+
+# Copy the current directory contents into the container at /app
+COPY . /app
+
+# Install dependencies
+RUN go mod download
+
+# Build with verbose output
+RUN go build -v -o ragtag
+
+EXPOSE 8080
+
+CMD ["./ragtag"]

docker-backup/docker-compose.yml

@@ -0,0 +1,19 @@
+version: '3.8'
+
+services:
+  ollama:
+    image: ollama/ollama:latest
+    container_name: ollama
+    ports:
+      - "11434:11434"  # Expose the default Ollama API port
+    environment:
+      - OLLAMA_PORT=11434
+      - OLLAMA_HOST=0.0.0.0
+      - OLLAMA_MODEL=llama3.1
+    restart: unless-stopped
+  api:
+    image: ragtag:latest
+    container_name: api
+    ports:
+      - "8080:8080"
+    restart: unless-stopped

docker-backup/requirements.txt

@@ -0,0 +1,2 @@
+flask
+ollama

docker-compose.yml

@@ -0,0 +1,51 @@
+services:
+  ollama:
+    build:
+      context: .
+      dockerfile: Dockerfile.ollama
+    container_name: ollama
+    ports:
+      - "11434:11434"
+    environment:
+      - OLLAMA_PORT=11434
+      - OLLAMA_HOST=0.0.0.0
+      - OLLAMA_MODEL=llama3.1
+    restart: unless-stopped
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost:11434/api/tags"]
+      interval: 30s
+      timeout: 10s
+      retries: 5
+
+  db:
+    image: pgvector/pgvector:pg16
+    container_name: db
+    ports:
+      - "5432:5432"
+    environment:
+      - POSTGRES_DB=ragtag
+      - POSTGRES_USER=${DB_USER}
+      - POSTGRES_PASSWORD=${DB_PASSWORD}
+    volumes:
+      - ./init-db.sql:/docker-entrypoint-initdb.d/init-db.sql
+    restart: unless-stopped
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U ${DB_USER} -d ragtag"]
+      interval: 10s
+      timeout: 5s
+      retries: 5
+
+  api:
+    image: ragtag:latest
+    container_name: api
+    ports:
+      - "8080:8080"
+    environment:
+      - DB_URL=${DB_URL_DOCKER}
+      - OLLAMA_HOST=ollama
+    restart: unless-stopped
+    depends_on:
+      db:
+        condition: service_healthy
+      ollama:
+        condition: service_healthy

docmanager.html

@@ -0,0 +1,134 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>Document Manager</title>
+    <style>
+        body {
+            font-family: 'Courier New', monospace;
+            background-color: #000;
+            color: #00ff00;
+            margin: 0;
+            padding: 20px;
+        }
+        h1 {
+            color: #00ff00;
+        }
+        table {
+            width: 100%;
+            border-collapse: collapse;
+        }
+        th, td {
+            border: 1px solid #00ff00;
+            padding: 8px;
+            text-align: left;
+        }
+        th {
+            background-color: #003300;
+        }
+        button {
+            background-color: #003300;
+            color: #00ff00;
+            border: 1px solid #00ff00;
+            padding: 5px 10px;
+            cursor: pointer;
+        }
+        button:hover {
+            background-color: #004400;
+        }
+        a {
+            color: #00ff00;
+            text-decoration: none;
+        }
+        a:hover {
+            text-decoration: underline;
+        }
+    </style>
+</head>
+<body>
+    <h1>Document Manager</h1>
+    <a href="/" style="margin-bottom: 20px; display: block;">Back to Chat</a>
+    <table id="docTable">
+        <thead>
+            <tr>
+                <th>Title</th>
+                <th>Count</th>
+                <th>Action</th>
+            </tr>
+        </thead>
+        <tbody>
+            <!-- Table rows will be dynamically added here -->
+        </tbody>
+    </table>
+
+    <h2>Upload New Document</h2>
+    <form id="uploadForm" enctype="multipart/form-data">
+        <input type="text" id="docTitle" name="title" placeholder="Document Title" required>
+        <input type="file" id="docFile" name="file" accept=".txt,.pdf,.jpg,.jpeg,.png" required>
+        <button type="submit">Upload</button>
+    </form>
+
+    <script>
+        async function fetchDocuments() {
+            let documents = [];
+            try {
+                documents = await fetch('/documents').then(r => r.json());
+                if (!Array.isArray(documents)) documents = [];
+            } catch {}
+            const tableBody = document.querySelector('#docTable tbody');
+            tableBody.innerHTML = '';
+            documents.forEach(doc => {
+                const row = tableBody.insertRow();
+                row.insertCell(0).textContent = doc.title;
+                row.insertCell(1).textContent = doc.count;
+                const deleteButton = document.createElement('button');
+                deleteButton.textContent = 'Delete';
+                deleteButton.onclick = () => deleteDocument(doc.title);
+                row.insertCell(2).appendChild(deleteButton);
+            });
+        }
+
+        async function deleteDocument(title) {
+            const response = await fetch('/delete_document', {
+                method: 'POST',
+                headers: {
+                    'Content-Type': 'application/json',
+                },
+                body: JSON.stringify({ title: title }),
+            });
+            if (response.ok) {
+                fetchDocuments();
+            } else {
+                alert('Error deleting document');
+            }
+        }
+
+        async function uploadDocument(formData) {
+            const response = await fetch('/upload_document', {
+                method: 'POST',
+                body: formData
+            });
+            if (response.ok) {
+                alert('Document uploaded successfully');
+                fetchDocuments();
+            } else {
+                let msg = 'Error uploading document';
+                try {
+                    const data = await response.json();
+                    if (data && data.error) msg = data.error;
+                } catch {}
+                alert(msg);
+            }
+        }
+
+        document.getElementById('uploadForm').addEventListener('submit', function(e) {
+            e.preventDefault();
+            const formData = new FormData(this);
+            uploadDocument(formData);
+        });
+
+        fetchDocuments();
+    </script>
+</body>
+</html>

docs/docs.go

@@ -0,0 +1,44 @@
+// Package docs Code generated by swaggo/swag. DO NOT EDIT
+package docs
+
+import "github.com/swaggo/swag"
+
+const docTemplate = `{
+    "schemes": {{ marshal .Schemes }},
+    "swagger": "2.0",
+    "info": {
+        "description": "{{escape .Description}}",
+        "title": "{{.Title}}",
+        "termsOfService": "http://swagger.io/terms/",
+        "contact": {
+            "name": "James Campbell",
+            "email": "[email protected]"
+        },
+        "license": {
+            "name": "Apache 2.0",
+            "url": "http://www.apache.org/licenses/LICENSE-2.0.html"
+        },
+        "version": "{{.Version}}"
+    },
+    "host": "{{.Host}}",
+    "basePath": "{{.BasePath}}",
+    "paths": {}
+}`
+
+// SwaggerInfo holds exported Swagger Info so clients can modify it
+var SwaggerInfo = &swag.Spec{
+	Version:          "1.0",
+	Host:             "localhost:8080",
+	BasePath:         "/",
+	Schemes:          []string{},
+	Title:            "RAGTAG API",
+	Description:      "This is the API for the RAGTAG system.",
+	InfoInstanceName: "swagger",
+	SwaggerTemplate:  docTemplate,
+	LeftDelim:        "{{",
+	RightDelim:       "}}",
+}
+
+func init() {
+	swag.Register(SwaggerInfo.InstanceName(), SwaggerInfo)
+}

docs/swagger.json

@@ -0,0 +1,20 @@
+{
+    "swagger": "2.0",
+    "info": {
+        "description": "This is the API for the RAGTAG system.",
+        "title": "RAGTAG API",
+        "termsOfService": "http://swagger.io/terms/",
+        "contact": {
+            "name": "James Campbell",
+            "email": "[email protected]"
+        },
+        "license": {
+            "name": "Apache 2.0",
+            "url": "http://www.apache.org/licenses/LICENSE-2.0.html"
+        },
+        "version": "1.0"
+    },
+    "host": "localhost:8080",
+    "basePath": "/",
+    "paths": {}
+}

docs/swagger.yaml

@@ -0,0 +1,15 @@
+basePath: /
+host: localhost:8080
+info:
+  contact:
+    email: [email protected]
+    name: James Campbell
+  description: This is the API for the RAGTAG system.
+  license:
+    name: Apache 2.0
+    url: http://www.apache.org/licenses/LICENSE-2.0.html
+  termsOfService: http://swagger.io/terms/
+  title: RAGTAG API
+  version: "1.0"
+paths: {}
+swagger: "2.0"

example.env

@@ -0,0 +1,4 @@
+DB_URL=postgres://postgres:password@localhost:5432/ragtag
+DB_URL_DOCKER=postgres://ragtag:ragtag@db:5432/ragtag
+DB_USER=ragtag
+DB_PASSWORD=ragtag

go.mod

@@ -0,0 +1,43 @@
+module github.com/james-see/ragtag4
+
+go 1.24.1
+
+require (
+	github.com/gin-gonic/gin v1.10.0
+	github.com/jackc/pgx/v5 v5.6.0
+	github.com/joho/godotenv v1.5.1
+	github.com/ledongthuc/pdf v0.0.0-20250511090121-5959a4027728
+	github.com/ollama/ollama v0.3.6
+	github.com/pgvector/pgvector-go v0.2.2
+)
+
+require (
+	github.com/bytedance/sonic v1.11.6 // indirect
+	github.com/bytedance/sonic/loader v0.1.1 // indirect
+	github.com/cloudwego/base64x v0.1.4 // indirect
+	github.com/cloudwego/iasm v0.2.0 // indirect
+	github.com/gabriel-vasile/mimetype v1.4.3 // indirect
+	github.com/gin-contrib/sse v0.1.0 // indirect
+	github.com/go-playground/locales v0.14.1 // indirect
+	github.com/go-playground/universal-translator v0.18.1 // indirect
+	github.com/go-playground/validator/v10 v10.20.0 // indirect
+	github.com/goccy/go-json v0.10.2 // indirect
+	github.com/jackc/pgpassfile v1.0.0 // indirect
+	github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
+	github.com/json-iterator/go v1.1.12 // indirect
+	github.com/klauspost/cpuid/v2 v2.2.7 // indirect
+	github.com/leodido/go-urn v1.4.0 // indirect
+	github.com/mattn/go-isatty v0.0.20 // indirect
+	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
+	github.com/modern-go/reflect2 v1.0.2 // indirect
+	github.com/pelletier/go-toml/v2 v2.2.2 // indirect
+	github.com/twitchyliquid64/golang-asm v0.15.1 // indirect
+	github.com/ugorji/go/codec v1.2.12 // indirect
+	golang.org/x/arch v0.8.0 // indirect
+	golang.org/x/crypto v0.25.0 // indirect
+	golang.org/x/net v0.25.0 // indirect
+	golang.org/x/sys v0.22.0 // indirect
+	golang.org/x/text v0.16.0 // indirect
+	google.golang.org/protobuf v1.34.1 // indirect
+	gopkg.in/yaml.v3 v3.0.1 // indirect
+)

go.sum

@@ -0,0 +1,152 @@
+entgo.io/ent v0.13.1 h1:uD8QwN1h6SNphdCCzmkMN3feSUzNnVvV/WIkHKMbzOE=
+entgo.io/ent v0.13.1/go.mod h1:qCEmo+biw3ccBn9OyL4ZK5dfpwg++l1Gxwac5B1206A=
+github.com/bytedance/sonic v1.11.6 h1:oUp34TzMlL+OY1OUWxHqsdkgC/Zfc85zGqw9siXjrc0=
+github.com/bytedance/sonic v1.11.6/go.mod h1:LysEHSvpvDySVdC2f87zGWf6CIKJcAvqab1ZaiQtds4=
+github.com/bytedance/sonic/loader v0.1.1 h1:c+e5Pt1k/cy5wMveRDyk2X4B9hF4g7an8N3zCYjJFNM=
+github.com/bytedance/sonic/loader v0.1.1/go.mod h1:ncP89zfokxS5LZrJxl5z0UJcsk4M4yY2JpfqGeCtNLU=
+github.com/cloudwego/base64x v0.1.4 h1:jwCgWpFanWmN8xoIUHa2rtzmkd5J2plF/dnLS6Xd/0Y=
+github.com/cloudwego/base64x v0.1.4/go.mod h1:0zlkT4Wn5C6NdauXdJRhSKRlJvmclQ1hhJgA0rcu/8w=
+github.com/cloudwego/iasm v0.2.0 h1:1KNIy1I1H9hNNFEEH3DVnI4UujN+1zjpuk6gwHLTssg=
+github.com/cloudwego/iasm v0.2.0/go.mod h1:8rXZaNYT2n95jn+zTI1sDr+IgcD2GVs0nlbbQPiEFhY=
+github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
+github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/gabriel-vasile/mimetype v1.4.3 h1:in2uUcidCuFcDKtdcBxlR0rJ1+fsokWf+uqxgUFjbI0=
+github.com/gabriel-vasile/mimetype v1.4.3/go.mod h1:d8uq/6HKRL6CGdk+aubisF/M5GcPfT7nKyLpA0lbSSk=
+github.com/gin-contrib/sse v0.1.0 h1:Y/yl/+YNO8GZSjAhjMsSuLt29uWRFHdHYUb5lYOV9qE=
+github.com/gin-contrib/sse v0.1.0/go.mod h1:RHrZQHXnP2xjPF+u1gW/2HnVO7nvIa9PG3Gm+fLHvGI=
+github.com/gin-gonic/gin v1.10.0 h1:nTuyha1TYqgedzytsKYqna+DfLos46nTv2ygFy86HFU=
+github.com/gin-gonic/gin v1.10.0/go.mod h1:4PMNQiOhvDRa013RKVbsiNwoyezlm2rm0uX/T7kzp5Y=
+github.com/go-pg/pg/v10 v10.11.0 h1:CMKJqLgTrfpE/aOVeLdybezR2om071Vh38OLZjsyMI0=
+github.com/go-pg/pg/v10 v10.11.0/go.mod h1:4BpHRoxE61y4Onpof3x1a2SQvi9c+q1dJnrNdMjsroA=
+github.com/go-pg/zerochecker v0.2.0 h1:pp7f72c3DobMWOb2ErtZsnrPaSvHd2W4o9//8HtF4mU=
+github.com/go-pg/zerochecker v0.2.0/go.mod h1:NJZ4wKL0NmTtz0GKCoJ8kym6Xn/EQzXRl2OnAe7MmDo=
+github.com/go-playground/assert/v2 v2.2.0 h1:JvknZsQTYeFEAhQwI4qEt9cyV5ONwRHC+lYKSsYSR8s=
+github.com/go-playground/assert/v2 v2.2.0/go.mod h1:VDjEfimB/XKnb+ZQfWdccd7VUvScMdVu0Titje2rxJ4=
+github.com/go-playground/locales v0.14.1 h1:EWaQ/wswjilfKLTECiXz7Rh+3BjFhfDFKv/oXslEjJA=
+github.com/go-playground/locales v0.14.1/go.mod h1:hxrqLVvrK65+Rwrd5Fc6F2O76J/NuW9t0sjnWqG1slY=
+github.com/go-playground/universal-translator v0.18.1 h1:Bcnm0ZwsGyWbCzImXv+pAJnYK9S473LQFuzCbDbfSFY=
+github.com/go-playground/universal-translator v0.18.1/go.mod h1:xekY+UJKNuX9WP91TpwSH2VMlDf28Uj24BCp08ZFTUY=
+github.com/go-playground/validator/v10 v10.20.0 h1:K9ISHbSaI0lyB2eWMPJo+kOS/FBExVwjEviJTixqxL8=
+github.com/go-playground/validator/v10 v10.20.0/go.mod h1:dbuPbCMFw/DrkbEynArYaCwl3amGuJotoKCe95atGMM=
+github.com/goccy/go-json v0.10.2 h1:CrxCmQqYDkv1z7lO7Wbh2HN93uovUHgrECaO5ZrCXAU=
+github.com/goccy/go-json v0.10.2/go.mod h1:6MelG93GURQebXPDq3khkgXZkazVtN9CRI+MGFi0w8I=
+github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
+github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
+github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
+github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
+github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
+github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
+github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
+github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo=
+github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM=
+github.com/jackc/pgx/v5 v5.6.0 h1:SWJzexBzPL5jb0GEsrPMLIsi/3jOo7RHlzTjcAeDrPY=
+github.com/jackc/pgx/v5 v5.6.0/go.mod h1:DNZ/vlrUnhWCoFGxHAG8U2ljioxukquj7utPDgtQdTw=
+github.com/jackc/puddle/v2 v2.2.1 h1:RhxXJtFG022u4ibrCSMSiu5aOq1i77R3OHKNJj77OAk=
+github.com/jackc/puddle/v2 v2.2.1/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
+github.com/jinzhu/inflection v1.0.0 h1:K317FqzuhWc8YvSVlFMCCUb36O/S9MCKRDI7QkRKD/E=
+github.com/jinzhu/inflection v1.0.0/go.mod h1:h+uFLlag+Qp1Va5pdKtLDYj+kHp5pxUVkryuEj+Srlc=
+github.com/jinzhu/now v1.1.5 h1:/o9tlHleP7gOFmsnYNz3RGnqzefHA47wQpKrrdTIwXQ=
+github.com/jinzhu/now v1.1.5/go.mod h1:d3SSVoowX0Lcu0IBviAWJpolVfI5UJVZZ7cO71lE/z8=
+github.com/jmoiron/sqlx v1.3.5 h1:vFFPA71p1o5gAeqtEAwLU4dnX2napprKtHr7PYIcN3g=
+github.com/jmoiron/sqlx v1.3.5/go.mod h1:nRVWtLre0KfCLJvgxzCsLVMogSvQ1zNJtpYr2Ccp0mQ=
+github.com/joho/godotenv v1.5.1 h1:7eLL/+HRGLY0ldzfGMeQkb7vMd0as4CfYvUVzLqw0N0=
+github.com/joho/godotenv v1.5.1/go.mod h1:f4LDr5Voq0i2e/R5DDNOoa2zzDfwtkZa6DnEwAbqwq4=
+github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM=
+github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo=
+github.com/klauspost/cpuid/v2 v2.0.9/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
+github.com/klauspost/cpuid/v2 v2.2.7 h1:ZWSB3igEs+d0qvnxR/ZBzXVmxkgt8DdzP6m9pfuVLDM=
+github.com/klauspost/cpuid/v2 v2.2.7/go.mod h1:Lcz8mBdAVJIBVzewtcLocK12l3Y+JytZYpaMropDUws=
+github.com/knz/go-libedit v1.10.1/go.mod h1:MZTVkCWyz0oBc7JOWP3wNAzd002ZbM/5hgShxwh4x8M=
+github.com/kr/pretty v0.3.0 h1:WgNl7dwNpEZ6jJ9k1snq4pZsg7DOEN8hP9Xw0Tsjwk0=
+github.com/kr/pretty v0.3.0/go.mod h1:640gp4NfQd8pI5XOwp5fnNeVWj67G7CFk/SaSQn7NBk=
+github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
+github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
+github.com/ledongthuc/pdf v0.0.0-20240201131950-da5b75280b06 h1:kacRlPN7EN++tVpGUorNGPn/4DnB7/DfTY82AOn6ccU=
+github.com/ledongthuc/pdf v0.0.0-20240201131950-da5b75280b06/go.mod h1:imJHygn/1yfhB7XSJJKlFZKl/J+dCPAknuiaGOshXAs=
+github.com/ledongthuc/pdf v0.0.0-20250511090121-5959a4027728 h1:QwWKgMY28TAXaDl+ExRDqGQltzXqN/xypdKP86niVn8=
+github.com/ledongthuc/pdf v0.0.0-20250511090121-5959a4027728/go.mod h1:1fEHWurg7pvf5SG6XNE5Q8UZmOwex51Mkx3SLhrW5B4=
+github.com/leodido/go-urn v1.4.0 h1:WT9HwE9SGECu3lg4d/dIA+jxlljEa1/ffXKmRjqdmIQ=
+github.com/leodido/go-urn v1.4.0/go.mod h1:bvxc+MVxLKB4z00jd1z+Dvzr47oO32F/QSNjSBOlFxI=
+github.com/lib/pq v1.10.9 h1:YXG7RB+JIjhP29X+OtkiDnYaXQwpS4JEWq7dtCCRUEw=
+github.com/lib/pq v1.10.9/go.mod h1:AlVN5x4E4T544tWzH6hKfbfQvm3HdbOxrmggDNAPY9o=
+github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
+github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
+github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
+github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
+github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
+github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M=
+github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk=
+github.com/ollama/ollama v0.3.6 h1:nA/N0AmjP327po5cZDGLqI40nl+aeei0pD0dLa92ypE=
+github.com/ollama/ollama v0.3.6/go.mod h1:YrWoNkFnPOYsnDvsf/Ztb1wxU9/IXrNsQHqcxbY2r94=
+github.com/pelletier/go-toml/v2 v2.2.2 h1:aYUidT7k73Pcl9nb2gScu7NSrKCSHIDE89b3+6Wq+LM=
+github.com/pelletier/go-toml/v2 v2.2.2/go.mod h1:1t835xjRzz80PqgE6HHgN2JOsmgYu/h4qDAS4n929Rs=
+github.com/pgvector/pgvector-go v0.2.2 h1:Q/oArmzgbEcio88q0tWQksv/u9Gnb1c3F1K2TnalxR0=
+github.com/pgvector/pgvector-go v0.2.2/go.mod h1:u5sg3z9bnqVEdpe1pkTij8/rFhTaMCMNyQagPDLK8gQ=
+github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
+github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
+github.com/rogpeppe/go-internal v1.9.0 h1:73kH8U+JUqXU8lRuOHeVHaa/SZPifC7BkcraZVejAe8=
+github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs=
+github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
+github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
+github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
+github.com/stretchr/objx v0.5.2/go.mod h1:FRsXN1f5AsAjCGJKqEizvkpNtU+EGNCLh3NxZ/8L+MA=
+github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
+github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
+github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
+github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
+github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
+github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
+github.com/stretchr/testify v1.9.0 h1:HtqpIVDClZ4nwg75+f6Lvsy/wHu+3BoSGCbBAcpTsTg=
+github.com/stretchr/testify v1.9.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
+github.com/tmthrgd/go-hex v0.0.0-20190904060850-447a3041c3bc h1:9lRDQMhESg+zvGYmW5DyG0UqvY96Bu5QYsTLvCHdrgo=
+github.com/tmthrgd/go-hex v0.0.0-20190904060850-447a3041c3bc/go.mod h1:bciPuU6GHm1iF1pBvUfxfsH0Wmnc2VbpgvbI9ZWuIRs=
+github.com/twitchyliquid64/golang-asm v0.15.1 h1:SU5vSMR7hnwNxj24w34ZyCi/FmDZTkS4MhqMhdFk5YI=
+github.com/twitchyliquid64/golang-asm v0.15.1/go.mod h1:a1lVb/DtPvCB8fslRZhAngC2+aY1QWCk3Cedj/Gdt08=
+github.com/ugorji/go/codec v1.2.12 h1:9LC83zGrHhuUA9l16C9AHXAqEV/2wBQ4nkvumAE65EE=
+github.com/ugorji/go/codec v1.2.12/go.mod h1:UNopzCgEMSXjBc6AOMqYvWC1ktqTAfzJZUZgYf6w6lg=
+github.com/uptrace/bun v1.1.12 h1:sOjDVHxNTuM6dNGaba0wUuz7KvDE1BmNu9Gqs2gJSXQ=
+github.com/uptrace/bun v1.1.12/go.mod h1:NPG6JGULBeQ9IU6yHp7YGELRa5Agmd7ATZdz4tGZ6z0=
+github.com/uptrace/bun/dialect/pgdialect v1.1.12 h1:m/CM1UfOkoBTglGO5CUTKnIKKOApOYxkcP2qn0F9tJk=
+github.com/uptrace/bun/dialect/pgdialect v1.1.12/go.mod h1:Ij6WIxQILxLlL2frUBxUBOZJtLElD2QQNDcu/PWDHTc=
+github.com/uptrace/bun/driver/pgdriver v1.1.12 h1:3rRWB1GK0psTJrHwxzNfEij2MLibggiLdTqjTtfHc1w=
+github.com/uptrace/bun/driver/pgdriver v1.1.12/go.mod h1:ssYUP+qwSEgeDDS1xm2XBip9el1y9Mi5mTAvLoiADLM=
+github.com/vmihailenco/bufpool v0.1.11 h1:gOq2WmBrq0i2yW5QJ16ykccQ4wH9UyEsgLm6czKAd94=
+github.com/vmihailenco/bufpool v0.1.11/go.mod h1:AFf/MOy3l2CFTKbxwt0mp2MwnqjNEs5H/UxrkA5jxTQ=
+github.com/vmihailenco/msgpack/v5 v5.3.5 h1:5gO0H1iULLWGhs2H5tbAHIZTV8/cYafcFOr9znI5mJU=
+github.com/vmihailenco/msgpack/v5 v5.3.5/go.mod h1:7xyJ9e+0+9SaZT0Wt1RGleJXzli6Q/V5KbhBonMG9jc=
+github.com/vmihailenco/tagparser v0.1.2 h1:gnjoVuB/kljJ5wICEEOpx98oXMWPLj22G67Vbd1qPqc=
+github.com/vmihailenco/tagparser v0.1.2/go.mod h1:OeAg3pn3UbLjkWt+rN9oFYB6u/cQgqMEUPoW2WPyhdI=
+github.com/vmihailenco/tagparser/v2 v2.0.0 h1:y09buUbR+b5aycVFQs/g70pqKVZNBmxwAhO7/IwNM9g=
+github.com/vmihailenco/tagparser/v2 v2.0.0/go.mod h1:Wri+At7QHww0WTrCBeu4J6bNtoV6mEfg5OIWRZA9qds=
+golang.org/x/arch v0.0.0-20210923205945-b76863e36670/go.mod h1:5om86z9Hs0C8fWVUuoMHwpExlXzs5Tkyp9hOrfG7pp8=
+golang.org/x/arch v0.8.0 h1:3wRIsP3pM4yUptoR96otTUOXI367OS0+c9eeRi9doIc=
+golang.org/x/arch v0.8.0/go.mod h1:FEVrYAQjsQXMVJ1nsMoVVXPZg6p2JE2mx8psSWTDQys=
+golang.org/x/crypto v0.25.0 h1:ypSNr+bnYL2YhwoMt2zPxHFmbAN1KZs/njMG3hxUp30=
+golang.org/x/crypto v0.25.0/go.mod h1:T+wALwcMOSE0kXgUAnPAHqTLW+XHgcELELW8VaDgm/M=
+golang.org/x/net v0.25.0 h1:d/OCCoBEUq33pjydKrGQhw7IlUPI2Oylr+8qLx49kac=
+golang.org/x/net v0.25.0/go.mod h1:JkAGAh7GEvH74S6FOH42FLoXpXbE/aqXSrIQjXgsiwM=
+golang.org/x/sync v0.7.0 h1:YsImfSBoP9QPYL0xyKJPq0gcaJdG3rInoqxTWbfQu9M=
+golang.org/x/sync v0.7.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
+golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.22.0 h1:RI27ohtqKCnwULzJLqkv897zojh5/DwS/ENaMzUOaWI=
+golang.org/x/sys v0.22.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
+golang.org/x/text v0.16.0 h1:a94ExnEXNtEwYLGJSIUxnWoxoRz/ZcCsV63ROupILh4=
+golang.org/x/text v0.16.0/go.mod h1:GhwF1Be+LQoKShO3cGOHzqOgRrGaYc9AvblQOmPVHnI=
+google.golang.org/protobuf v1.34.1 h1:9ddQBjfCyZPOHPUiPxpYESBLc+T8P3E+Vo4IbKZgFWg=
+google.golang.org/protobuf v1.34.1/go.mod h1:c6P6GXX6sHbq/GpV6MGZEdwhWPcYBgnhAHhKbcUYpos=
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
+gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
+gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
+gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
+gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
+gorm.io/driver/postgres v1.5.4 h1:Iyrp9Meh3GmbSuyIAGyjkN+n9K+GHX9b9MqsTL4EJCo=
+gorm.io/driver/postgres v1.5.4/go.mod h1:Bgo89+h0CRcdA33Y6frlaHHVuTdOf87pmyzwW9C/BH0=
+gorm.io/gorm v1.25.5 h1:zR9lOiiYf09VNh5Q1gphfyia1JpiClIWG9hQaxB/mls=
+gorm.io/gorm v1.25.5/go.mod h1:hbnx/Oo0ChWMn1BIhpy1oYozzpM15i4YPuHDmfYtwg8=
+mellium.im/sasl v0.3.1 h1:wE0LW6g7U83vhvxjC1IY8DnXM+EU095yeo8XClvCdfo=
+mellium.im/sasl v0.3.1/go.mod h1:xm59PUYpZHhgQ9ZqoJ5QaCqzWMi8IeS49dhp6plPCzw=
+nullprogram.com/x/optparse v1.0.0/go.mod h1:KdyPE+Igbe0jQUrVfMqDMeJQIJZEuyV7pjYmp6pbG50=
+rsc.io/pdf v0.1.1/go.mod h1:n8OzWcQ6Sp37PL01nO98y4iUCRdTGarVfzxY20ICaU4=

index.html

@@ -0,0 +1,227 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>RAGTAG MVP</title>
+    <link rel="icon" type="image/x-icon" href="/favicon.ico">
+    <style>
+        body {
+            font-family: 'Courier New', monospace;
+            background-color: #000;
+            color: #00ff00;
+            margin: 0;
+            padding: 0;
+            height: 100vh;
+            display: flex;
+            flex-direction: column;
+        }
+        #content {
+            flex-grow: 1;
+            overflow-y: auto;
+            padding: 20px;
+        }
+        #chat-container {
+            border: 1px solid #00ff00;
+            padding: 10px;
+            margin-bottom: 10px;
+            min-height: 200px;
+        }
+        #input-container {
+            display: flex;
+            padding: 10px 20px;
+            background-color: #000;
+            border-top: 1px solid #00ff00;
+        }
+        #prompt {
+            color: #00ff00;
+            margin-right: 5px;
+        }
+        #user-input {
+            flex-grow: 1;
+            background-color: #000;
+            border: none;
+            color: #00ff00;
+            font-family: 'Courier New', monospace;
+            font-size: 16px;
+        }
+        #user-input:focus {
+            outline: none;
+        }
+        .message {
+            margin-bottom: 10px;
+        }
+        .user-message {
+            color: #ffffff;
+        }
+        .ai-message {
+            color: #00ff00;
+        }
+    </style>
+</head>
+<body>
+    <div id="content">
+        <a href="/docmanager" style="color: #00ff00; margin-bottom: 10px; display: block;">Document Manager</a>
+        <a href="/describer" style="color: #00ff00; margin-bottom: 10px; display: block;">Image Describer</a>
+        <div id="sessionInfo" style="color: #00ff00; margin-bottom: 10px;">
+            <div>Session ID: <span id="sessionIdDisplay"></span></div>
+            <div>Title Filter: <span id="titleFilterDisplay">None</span></div>
+        </div>
+        <button id="clearButton" style="background-color: #003300; color: #00ff00; border: 1px solid #00ff00; padding: 5px 10px; cursor: pointer; margin-bottom: 10px;">Clear Chat</button>
+        <div id="chat-container"></div>
+    </div>
+    <div id="input-container">
+        <span id="prompt">$</span>
+        <input type="text" id="user-input" placeholder="Enter your query...">
+    </div>
+
+    <script>
+        const chatContainer = document.getElementById('chat-container');
+        const userInput = document.getElementById('user-input');
+        const clearButton = document.getElementById('clearButton');
+        const sessionIdDisplay = document.getElementById('sessionIdDisplay');
+        const titleFilterDisplay = document.getElementById('titleFilterDisplay');
+        let queryHistory = [];
+        let historyIndex = -1;
+        let sessionId = Date.now().toString();
+        let titleFilter = '';
+
+        // Display initial session ID
+        sessionIdDisplay.textContent = sessionId;
+
+        function addMessage(sender, message) {
+            const messageElement = document.createElement('div');
+            messageElement.classList.add('message');
+            messageElement.classList.add(sender === 'You' ? 'user-message' : 'ai-message');
+            messageElement.textContent = `${sender === 'You' ? '$ ' : ''}${message}`;
+            chatContainer.appendChild(messageElement);
+            chatContainer.scrollTop = chatContainer.scrollHeight;
+        }
+
+        async function sendQuery(query) {
+            try {
+                const response = await fetch('/query', {
+                    method: 'POST',
+                    headers: {
+                        'Content-Type': 'application/json',
+                    },
+                    body: JSON.stringify({ query: query, sessionId: sessionId, titleFilter: titleFilter }),
+                });
+
+                if (!response.ok) {
+                    throw new Error('Network response was not ok');
+                }
+
+                const reader = response.body.getReader();
+                addMessage('AI', ''); // Add an empty AI message to start
+
+                let buffer = '';
+                while (true) {
+                    const { value, done } = await reader.read();
+                    if (done) break;
+
+                    buffer += new TextDecoder().decode(value);
+                    const lines = buffer.split('\n');
+                    buffer = lines.pop() || '';
+
+                    for (const line of lines) {
+                        if (line.startsWith('data:')) {
+                            const content = line.slice(5);
+                            if (content) {
+                                appendToLastAIMessage(content);
+                            } else {
+                                appendToLastAIMessage('\n');
+                            }
+                        }
+                    }
+                }
+            } catch (error) {
+                console.error('Error:', error);
+                addMessage('System', 'An error occurred while processing your request.');
+            }
+        }
+
+        function appendToLastAIMessage(content) {
+            const messages = document.querySelectorAll('.message');
+            const lastAIMessage = Array.from(messages).reverse().find(msg => msg.classList.contains('ai-message'));
+            if (lastAIMessage) {
+                if (content === '\n') {
+                    lastAIMessage.appendChild(document.createElement('br'));
+                } else {
+                    lastAIMessage.appendChild(document.createTextNode(content));
+                }
+                lastAIMessage.scrollIntoView({ behavior: 'smooth', block: 'end' });
+            } else {
+                addMessage('AI', content);
+            }
+        }
+
+        async function clearSession() {
+            try {
+                const response = await fetch('/clear_session', {
+                    method: 'POST',
+                    headers: {
+                        'Content-Type': 'application/json',
+                    },
+                    body: JSON.stringify({ sessionId: sessionId }),
+                });
+
+                if (!response.ok) {
+                    throw new Error('Network response was not ok');
+                }
+
+                chatContainer.innerHTML = '';
+                queryHistory = [];
+                historyIndex = -1;
+                sessionId = Date.now().toString();
+                titleFilter = '';
+                sessionIdDisplay.textContent = sessionId;
+                titleFilterDisplay.textContent = 'None';
+                addMessage('System', 'Chat session cleared.');
+            } catch (error) {
+                console.error('Error:', error);
+                addMessage('System', 'An error occurred while clearing the session.');
+            }
+        }
+
+        function processQuery(query) {
+            if (query.includes('@')) {
+                const parts = query.split('@');
+                if (parts.length > 1) {
+                    titleFilter = parts[1].split(' ')[0];
+                    titleFilterDisplay.textContent = titleFilter;
+                }
+            }
+            return query;
+        }
+
+        userInput.addEventListener('keydown', (e) => {
+            if (e.key === 'Enter') {
+                const query = userInput.value.trim();
+                if (query) {
+                    addMessage('You', query);
+                    const processedQuery = processQuery(query);
+                    sendQuery(processedQuery);
+                    queryHistory.unshift(query);
+                    historyIndex = -1;
+                    userInput.value = '';
+                }
+            } else if (e.key === 'ArrowUp') {
+                e.preventDefault();
+                if (historyIndex < queryHistory.length - 1) {
+                    historyIndex++;
+                    userInput.value = queryHistory[historyIndex];
+                }
+            } else if (e.key === 'ArrowDown') {
+                e.preventDefault();
+                if (historyIndex > -1) {
+                    historyIndex--;
+                    userInput.value = historyIndex === -1 ? '' : queryHistory[historyIndex];
+                }
+            }
+        });
+
+        clearButton.addEventListener('click', clearSession);
+    </script>
+</body>
+</html>

init-db.sql

@@ -0,0 +1,8 @@
+CREATE EXTENSION IF NOT EXISTS vector;
+
+CREATE TABLE IF NOT EXISTS items (
+    id SERIAL PRIMARY KEY,
+    title TEXT,
+    doc TEXT,
+    embedding vector(4096)
+);

main.go

@@ -0,0 +1,698 @@
+package main
+
+import (
+	"context"
+	"fmt"
+	"io"
+	"log"
+	"net/http"
+	"net/url"
+	"os"
+	"path/filepath"
+	"strings"
+
+	got "github.com/joho/godotenv"
+
+	"bytes"
+	"encoding/base64"
+	"encoding/json"
+
+	"unicode/utf8"
+
+	"github.com/gin-gonic/gin"
+	"github.com/jackc/pgx/v5"
+	"github.com/ledongthuc/pdf"
+	"github.com/ollama/ollama/api"
+	"github.com/pgvector/pgvector-go"
+)
+
+// @title RAGTAG API
+// @version 1.0
+// @description This is the API for the RAGTAG system.
+// @termsOfService http://swagger.io/terms/
+
+// @contact.name James Campbell
+// @contact.email [email protected]
+
+// @license.name Apache 2.0
+// @license.url http://www.apache.org/licenses/LICENSE-2.0.html
+
+// @host localhost:8080
+// @BasePath /
+
+type Session struct {
+	Messages    []api.Message
+	TitleFilter string
+}
+
+var sessions = make(map[string]*Session)
+
+func generateEmbedding(input string) ([]float32, error) {
+	ollamaHost := os.Getenv("OLLAMA_HOST")
+	if ollamaHost == "" {
+		ollamaHost = "localhost" // fallback to localhost if not set
+	}
+	ollamaURL, err := url.Parse(fmt.Sprintf("http://%s:11434", ollamaHost))
+	if err != nil {
+		return nil, err
+	}
+	client := api.NewClient(ollamaURL, http.DefaultClient)
+
+	// Create an embedding request
+	req := &api.EmbedRequest{
+		Model: "llama3.1", // Ensure this is an embedding-capable model
+		Input: input,
+	}
+
+	// Call the Embed function
+	resp, err := client.Embed(context.Background(), req)
+	if err != nil {
+		return nil, err
+	}
+
+	return resp.Embeddings[0], nil
+}
+
+func insertItem(conn *pgx.Conn, title string, docText string, embedding []float32) error {
+	// Combine title and docText for embedding
+	combinedText := title + " " + docText
+
+	_, err := conn.Exec(context.Background(),
+		"INSERT INTO items (title, doc, embedding) VALUES ($1, $2, $3)",
+		title, combinedText, pgvector.NewVector(embedding))
+
+	return err
+}
+
+func queryEmbeddings(conn *pgx.Conn, query string, session *Session, c *gin.Context) error {
+	// Generate embedding for the query
+	queryEmbedding, err := generateEmbedding(query)
+	if err != nil {
+		return err
+	}
+
+	// Prepare the SQL query
+	sqlQuery := "SELECT doc, COALESCE(title, 'Untitled') FROM items"
+	if session.TitleFilter != "" {
+		sqlQuery += fmt.Sprintf(" WHERE title LIKE '%%%s%%'", session.TitleFilter)
+	}
+	sqlQuery += " ORDER BY embedding <-> $1 LIMIT 5"
+
+	// Query the database for similar documents
+	rows, err := conn.Query(context.Background(), sqlQuery, pgvector.NewVector(queryEmbedding))
+	if err != nil {
+		return err
+	}
+	defer rows.Close()
+
+	var docs []string
+	var sources []string
+	for rows.Next() {
+		var doc, title string
+		if err := rows.Scan(&doc, &title); err != nil {
+			return err
+		}
+		docs = append(docs, doc)
+		sources = append(sources, fmt.Sprintf("Source: %s", title))
+	}
+
+	// Combine the retrieved documents
+	contextText := strings.Join(docs, "\n\n")
+
+	// Create a chat request
+	ollamaHost := os.Getenv("OLLAMA_HOST")
+	if ollamaHost == "" {
+		ollamaHost = "localhost" // fallback to localhost if not set
+	}
+	ollamaURL, err := url.Parse(fmt.Sprintf("http://%s:11434", ollamaHost))
+	if err != nil {
+		return err
+	}
+	client := api.NewClient(ollamaURL, http.DefaultClient)
+
+	// Add the new query to the session
+	session.Messages = append(session.Messages, api.Message{Role: "user", Content: query})
+
+	// Prepare the messages for the chat request
+	messages := []api.Message{
+		{Role: "system", Content: "You are an assistant that answers questions based on the given context."},
+		{Role: "user", Content: "Here's the context:\n" + contextText},
+	}
+	messages = append(messages, session.Messages...)
+
+	req := &api.ChatRequest{
+		Model:    "llama3.1",
+		Messages: messages,
+		Stream:   new(bool), // Use new(bool) to create a pointer to a boolean
+	}
+	*req.Stream = true // Set the value to true
+
+	// Call the Chat function with streaming
+	err = client.Chat(context.Background(), req, func(resp api.ChatResponse) error {
+		// Send the raw content without any modifications
+		if resp.Message.Content != "" {
+			c.SSEvent("message", resp.Message.Content)
+			c.Writer.Flush() // Ensure the content is sent immediately
+		}
+		return nil
+	})
+	if err != nil {
+		return err
+	}
+
+	// Add the AI response to the session
+	session.Messages = append(session.Messages, api.Message{Role: "assistant", Content: "Response sent via streaming"})
+
+	return nil
+}
+
+func getDocuments(conn *pgx.Conn) ([]map[string]interface{}, error) {
+	rows, err := conn.Query(context.Background(), "SELECT DISTINCT ON (SPLIT_PART(title, '_chunk_', 1)) SPLIT_PART(title, '_chunk_', 1) as title, COUNT(*) as count FROM items GROUP BY SPLIT_PART(title, '_chunk_', 1)")
+	if err != nil {
+		return nil, err
+	}
+	defer rows.Close()
+
+	var documents []map[string]interface{}
+	for rows.Next() {
+		var title string
+		var count int
+		if err := rows.Scan(&title, &count); err != nil {
+			return nil, err
+		}
+		documents = append(documents, map[string]interface{}{
+			"title": title,
+			"count": count,
+		})
+	}
+
+	return documents, nil
+}
+
+func deleteDocument(conn *pgx.Conn, title string) error {
+	_, err := conn.Exec(context.Background(), "DELETE FROM items WHERE title LIKE $1 || '%'", title)
+	return err
+}
+
+func uploadDocument(c *gin.Context, conn *pgx.Conn) {
+	title := c.PostForm("title")
+	file, header, err := c.Request.FormFile("file")
+	if err != nil {
+		log.Printf("Error getting file: %v", err)
+		c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
+		return
+	}
+	defer file.Close()
+
+	// Create uploads directory if it doesn't exist
+	uploadsDir := "uploads"
+	if err := os.MkdirAll(uploadsDir, 0755); err != nil {
+		log.Printf("Error creating uploads directory: %v", err)
+		c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to create uploads directory"})
+		return
+	}
+
+	filename := filepath.Join(uploadsDir, header.Filename)
+	out, err := os.Create(filename)
+	if err != nil {
+		log.Printf("Error creating file: %v", err)
+		c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
+		return
+	}
+	defer out.Close()
+
+	_, err = io.Copy(out, file)
+	if err != nil {
+		log.Printf("Error copying file: %v", err)
+		c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
+		return
+	}
+
+	// Debug: log file size and first 16 bytes
+	stat, statErr := os.Stat(filename)
+	if statErr == nil {
+		log.Printf("Uploaded file size: %d bytes", stat.Size())
+		fcheck, ferr := os.Open(filename)
+		if ferr == nil {
+			buf := make([]byte, 16)
+			n, _ := fcheck.Read(buf)
+			log.Printf("First 16 bytes: % x", buf[:n])
+			fcheck.Close()
+		}
+	}
+
+	var textContent string
+	ext := strings.ToLower(filepath.Ext(filename))
+	if ext == ".jpg" || ext == ".jpeg" || ext == ".png" {
+		// Generate image summary using the llava model
+		summary, err := generateImageSummary(filename)
+		if err != nil {
+			log.Printf("Error generating image summary: %v", err)
+			c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
+			return
+		}
+		textContent = summary
+	} else if ext == ".pdf" {
+		// Check PDF signature before parsing
+		f, rErr := os.Open(filename)
+		if rErr != nil {
+			log.Printf("Error opening PDF: %v", rErr)
+			c.JSON(http.StatusInternalServerError, gin.H{"error": rErr.Error()})
+			return
+		}
+		defer f.Close()
+		buf := make([]byte, 5)
+		_, err := f.Read(buf)
+		if err != nil || string(buf) != "%PDF-" {
+			log.Printf("Uploaded file is not a valid PDF (missing %PDF- header): %s", filename)
+			c.JSON(http.StatusBadRequest, gin.H{"error": "Uploaded file is not a valid PDF (missing %PDF- header)"})
+			return
+		}
+		stat, _ := f.Stat()
+		// Loosen EOF check: search last 1KB for %%EOF
+		eofCheckSize := int64(1024)
+		if stat.Size() < eofCheckSize {
+			eofCheckSize = stat.Size()
+		}
+		endBuf := make([]byte, eofCheckSize)
+		_, err = f.ReadAt(endBuf, stat.Size()-eofCheckSize)
+		if err != nil || !strings.Contains(string(endBuf), "%%EOF") {
+			log.Printf("Uploaded file is not a valid PDF (missing %%EOF): %s", filename)
+			c.JSON(http.StatusBadRequest, gin.H{"error": "Uploaded file is not a valid PDF (missing %%EOF)"})
+			return
+		}
+		// Reset file pointer for pdf.NewReader
+		f.Seek(0, 0)
+		reader, pdfErr := pdf.NewReader(f, stat.Size())
+		if pdfErr != nil {
+			log.Printf("Error reading PDF: %v", pdfErr)
+			c.JSON(http.StatusBadRequest, gin.H{"error": "Uploaded file is not a valid PDF or is corrupted"})
+			return
+		}
+		var sb strings.Builder
+		for i := 1; i <= reader.NumPage(); i++ {
+			page := reader.Page(i)
+			if page.V.IsNull() {
+				continue
+			}
+			content, err := page.GetPlainText(nil)
+			if err != nil {
+				log.Printf("Error extracting text from page %d: %v", i, err)
+				continue
+			}
+			sb.WriteString(content)
+		}
+		textContent = sb.String()
+		log.Printf("Extracted text length: %d", len(textContent))
+	} else {
+		content, err := os.ReadFile(filename)
+		if err != nil {
+			log.Printf("Error reading file: %v", err)
+			c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
+			return
+		}
+		textContent = string(content)
+	}
+
+	// Remove null bytes (Postgres TEXT cannot contain 0x00)
+	textContent = strings.ReplaceAll(textContent, "\x00", "")
+
+	// Validate UTF-8
+	if !utf8.ValidString(textContent) {
+		log.Printf("Invalid UTF-8 detected in document: %s", filename)
+		c.JSON(http.StatusBadRequest, gin.H{"error": "Uploaded document is not valid UTF-8"})
+		return
+	}
+
+	// Generate embedding for the text content using llama3.1
+	embedding, err := generateEmbedding(textContent)
+	if err != nil {
+		log.Printf("Error generating embedding: %v", err)
+		c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
+		return
+	}
+
+	// Insert the document into the database
+	err = insertItem(conn, title, textContent, embedding)
+	if err != nil {
+		log.Printf("Error inserting item: %v", err)
+		c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
+		return
+	}
+
+	c.JSON(http.StatusOK, gin.H{"message": "Document uploaded and processed successfully"})
+}
+
+func chunkText(text string, chunkSize int) []string {
+	words := strings.Fields(text)
+	var chunks []string
+	for i := 0; i < len(words); i += chunkSize {
+		end := i + chunkSize
+		if end > len(words) {
+			end = len(words)
+		}
+		chunks = append(chunks, strings.Join(words[i:end], " "))
+	}
+	return chunks
+}
+
+func generateImageSummary(imagePath string) (string, error) {
+	imageData, err := os.ReadFile(imagePath)
+	if err != nil {
+		return "", fmt.Errorf("failed to read image file: %w", err)
+	}
+
+	base64Image := base64.StdEncoding.EncodeToString(imageData)
+
+	payload := map[string]interface{}{
+		"model":  "llava",
+		"prompt": "Describe this image in detail:",
+		"images": []string{base64Image},
+		"stream": true,
+	}
+
+	jsonPayload, err := json.Marshal(payload)
+	if err != nil {
+		return "", fmt.Errorf("failed to marshal JSON payload: %w", err)
+	}
+
+	ollamaHost := os.Getenv("OLLAMA_HOST")
+	if ollamaHost == "" {
+		ollamaHost = "localhost"
+	}
+	url := fmt.Sprintf("http://%s:11434/api/generate", ollamaHost)
+
+	resp, err := http.Post(url, "application/json", bytes.NewBuffer(jsonPayload))
+	if err != nil {
+		return "", fmt.Errorf("failed to send POST request: %w", err)
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode != http.StatusOK {
+		body, _ := io.ReadAll(resp.Body)
+		return "", fmt.Errorf("unexpected response status: %d, body: %s", resp.StatusCode, string(body))
+	}
+
+	var summary strings.Builder
+	decoder := json.NewDecoder(resp.Body)
+	for {
+		var result struct {
+			Response string `json:"response"`
+			Done     bool   `json:"done"`
+		}
+		if err := decoder.Decode(&result); err != nil {
+			if err == io.EOF {
+				break
+			}
+			return "", fmt.Errorf("failed to decode JSON response: %w", err)
+		}
+		summary.WriteString(result.Response)
+		if result.Done {
+			break
+		}
+	}
+
+	if summary.Len() == 0 {
+		return "", fmt.Errorf("empty response from llava model")
+	}
+
+	fmt.Println("The summary of the image is: ", summary.String())
+
+	return summary.String(), nil
+}
+
+func main() {
+	// Set up the database connection
+	// load env variables
+	got.Load()
+	conn, err := pgx.Connect(context.Background(), os.Getenv("DB_URL"))
+	if err != nil {
+		log.Fatal("Unable to connect to database:", err)
+	}
+	defer conn.Close(context.Background())
+
+	// Set up the Gin router
+	r := gin.Default()
+
+	// Define the /add_document endpoint
+	r.POST("/add_document", func(c *gin.Context) {
+		var request struct {
+			Title   string `json:"title"`
+			DocText string `json:"doc_text"`
+		}
+		if err := c.BindJSON(&request); err != nil {
+			c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
+			return
+		}
+
+		// Generate the embedding
+		embedding, err := generateEmbedding(request.DocText)
+		if err != nil {
+			c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
+			return
+		}
+
+		// Insert the document and its embedding into the items table
+		err = insertItem(conn, request.Title, request.DocText, embedding)
+		if err != nil {
+			c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
+			return
+		}
+
+		c.JSON(http.StatusOK, gin.H{"message": "Document chunk embedded and stored successfully!"})
+	})
+
+	// Add the new /query endpoint
+	r.POST("/query", func(c *gin.Context) {
+		var request struct {
+			Query     string `json:"query"`
+			SessionID string `json:"sessionId"`
+		}
+		if err := c.BindJSON(&request); err != nil {
+			c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
+			return
+		}
+
+		session, ok := sessions[request.SessionID]
+		if !ok {
+			session = &Session{
+				Messages: []api.Message{
+					{Role: "system", Content: "You are an assistant that answers questions based on the given context."},
+				},
+				TitleFilter: "",
+			}
+			sessions[request.SessionID] = session
+		}
+
+		// Check for @title in the query
+		if strings.Contains(request.Query, "@") {
+			parts := strings.Split(request.Query, "@")
+			if len(parts) > 1 {
+				session.TitleFilter = strings.Split(parts[1], " ")[0]
+				request.Query = strings.Replace(request.Query, "@"+session.TitleFilter, "", 1)
+			}
+		}
+
+		c.Header("Content-Type", "text/event-stream")
+		c.Header("Cache-Control", "no-cache")
+		c.Header("Connection", "keep-alive")
+		c.Header("Access-Control-Allow-Origin", "*")
+		c.Header("Access-Control-Allow-Credentials", "true")
+		c.Header("Access-Control-Allow-Headers", "Content-Type")
+		c.Header("Access-Control-Allow-Methods", "POST")
+		c.Header("encoding", "chunked")
+
+		err := queryEmbeddings(conn, request.Query, session, c)
+		if err != nil {
+			c.SSEvent("error", err.Error())
+		}
+		c.SSEvent("done", "")
+	})
+
+	// Serve the index.html file
+	r.GET("/", func(c *gin.Context) {
+		c.File("index.html")
+	})
+
+	// Serve the docmanager.html file
+	r.GET("/docmanager", func(c *gin.Context) {
+		c.File("docmanager.html")
+	})
+
+	// Add a new endpoint to fetch documents
+	r.GET("/documents", func(c *gin.Context) {
+		documents, err := getDocuments(conn)
+		if err != nil {
+			c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
+			return
+		}
+		c.JSON(http.StatusOK, documents)
+	})
+
+	// Add a new endpoint to delete documents
+	r.POST("/delete_document", func(c *gin.Context) {
+		var request struct {
+			Title string `json:"title"`
+		}
+		if err := c.BindJSON(&request); err != nil {
+			c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
+			return
+		}
+
+		err := deleteDocument(conn, request.Title)
+		if err != nil {
+			c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
+			return
+		}
+
+		c.JSON(http.StatusOK, gin.H{"message": "Document deleted successfully"})
+	})
+
+	// Add a new endpoint to upload documents
+	r.POST("/upload_document", func(c *gin.Context) {
+		uploadDocument(c, conn)
+	})
+
+	// Add a new endpoint to clear the chat session
+	r.POST("/clear_session", func(c *gin.Context) {
+		var request struct {
+			SessionID string `json:"sessionId"`
+		}
+		if err := c.BindJSON(&request); err != nil {
+			c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
+			return
+		}
+
+		delete(sessions, request.SessionID)
+		c.JSON(http.StatusOK, gin.H{"message": "Chat session cleared successfully"})
+	})
+
+	// Add a new endpoint to check if Twitter data exists
+	r.GET("/check_data", func(c *gin.Context) {
+		rows, err := conn.Query(context.Background(), "SELECT DISTINCT title FROM items WHERE title LIKE '%twitter%'")
+		if err != nil {
+			c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
+			return
+		}
+		defer rows.Close()
+
+		var titles []string
+		for rows.Next() {
+			var title string
+			if err := rows.Scan(&title); err != nil {
+				c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
+				return
+			}
+			titles = append(titles, title)
+		}
+
+		c.JSON(http.StatusOK, gin.H{"twitter_titles": titles})
+	})
+
+	// Serve the describer.html file
+	r.GET("/describer", func(c *gin.Context) {
+		c.File("describer.html")
+	})
+
+	// Handle image description
+	r.POST("/describe_image", func(c *gin.Context) {
+		file, _, err := c.Request.FormFile("file")
+		if err != nil {
+			c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
+			return
+		}
+		defer file.Close()
+
+		// Create a temporary file to store the uploaded image
+		tempFile, err := os.CreateTemp("", "uploaded-*.jpg")
+		if err != nil {
+			c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
+			return
+		}
+		defer os.Remove(tempFile.Name())
+		defer tempFile.Close()
+
+		// Copy the uploaded file to the temporary file
+		_, err = io.Copy(tempFile, file)
+		if err != nil {
+			c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
+			return
+		}
+
+		c.Header("Content-Type", "text/event-stream")
+		c.Header("Cache-Control", "no-cache")
+		c.Header("Connection", "keep-alive")
+		c.Header("Access-Control-Allow-Origin", "*")
+		c.Header("Access-Control-Allow-Credentials", "true")
+		c.Header("Access-Control-Allow-Headers", "Content-Type")
+		c.Header("Access-Control-Allow-Methods", "POST")
+		c.Header("encoding", "chunked")
+
+		imageData, err := os.ReadFile(tempFile.Name())
+		if err != nil {
+			c.SSEvent("error", err.Error())
+			return
+		}
+
+		base64Image := base64.StdEncoding.EncodeToString(imageData)
+
+		payload := map[string]interface{}{
+			"model":  "llava",
+			"prompt": "Describe this image in detail:",
+			"images": []string{base64Image},
+			"stream": true,
+		}
+
+		jsonPayload, err := json.Marshal(payload)
+		if err != nil {
+			c.SSEvent("error", err.Error())
+			return
+		}
+
+		ollamaHost := os.Getenv("OLLAMA_HOST")
+		if ollamaHost == "" {
+			ollamaHost = "localhost"
+		}
+		url := fmt.Sprintf("http://%s:11434/api/generate", ollamaHost)
+
+		resp, err := http.Post(url, "application/json", bytes.NewBuffer(jsonPayload))
+		if err != nil {
+			c.SSEvent("error", err.Error())
+			return
+		}
+		defer resp.Body.Close()
+
+		if resp.StatusCode != http.StatusOK {
+			body, _ := io.ReadAll(resp.Body)
+			c.SSEvent("error", fmt.Sprintf("Unexpected response status: %d, body: %s", resp.StatusCode, string(body)))
+			return
+		}
+
+		decoder := json.NewDecoder(resp.Body)
+		for {
+			var result struct {
+				Response string `json:"response"`
+				Done     bool   `json:"done"`
+			}
+			if err := decoder.Decode(&result); err != nil {
+				if err == io.EOF {
+					break
+				}
+				c.SSEvent("error", err.Error())
+				return
+			}
+			if result.Response != "" {
+				c.SSEvent("message", result.Response)
+				c.Writer.Flush() // Ensure the content is sent immediately
+			}
+			if result.Done {
+				break
+			}
+		}
+
+		c.SSEvent("done", "")
+	})
+
+	// Run the Gin server
+	r.Run(":8080")
+}

ollama-entrypoint.sh

@@ -0,0 +1,17 @@
+#!/bin/bash
+set -e
+
+# Start Ollama in the background
+ollama serve &
+
+# Wait for Ollama to be ready
+until curl -s -o /dev/null -w "%{http_code}" http://localhost:11434/api/tags | grep -q "200"; do
+  echo "Waiting for Ollama to be ready..."
+  sleep 5
+done
+
+# Pull the llama3.1 model
+ollama pull llama3.1
+ollama pull llava
+# Keep the container running
+wait

ragtag4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:85c0b559c6b2545fe5174963fd47313c490dda984c16bbd4db0d2b8c9eaa56dd
+size 18022930

screenplays/downloader.py

@@ -0,0 +1,63 @@
+import os
+import sys
+import requests
+from bs4 import BeautifulSoup
+import argparse
+
+# Default URL of the script to download
+DEFAULT_SCRIPT_URL = "https://www.dailyscript.com/scripts/twelve_monkeys.html"
+
+# Directory to save the downloaded script
+save_dir = "scripts"
+os.makedirs(save_dir, exist_ok=True)
+
+# Function to download a script
+def download_script(script_url, save_dir):
+    response = requests.get(script_url)
+    response.raise_for_status()
+
+    # Parse the HTML content
+    soup = BeautifulSoup(response.text, "html.parser")
+
+    # Try to find the <pre> tag
+    pre_tag = soup.find("pre")
+
+    if pre_tag:
+        # Extract the text content from the <pre> tag
+        text_content = pre_tag.get_text()
+        # Remove the line numbers
+        lines = text_content.split("\n")
+        cleaned_lines = [line.split("|", 1)[-1] for line in lines]
+        cleaned_text = "\n".join(cleaned_lines)
+    else:
+        # If no <pre> tag, get the text from the body
+        body = soup.find("body")
+        if body:
+            cleaned_text = body.get_text()
+        else:
+            raise ValueError("Could not find script content in the HTML")
+
+    # Extract the filename from the URL and change the extension to .txt
+    filename = os.path.basename(script_url).split(".")[0] + ".txt"
+    save_path = os.path.join(save_dir, filename)
+
+    with open(save_path, "w", encoding="utf-8") as file:
+        file.write(cleaned_text)
+
+    print(f"Downloaded: {save_path}")
+    return save_path
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(description="Download a screenplay from a given URL.")
+    parser.add_argument("--url", type=str, default=DEFAULT_SCRIPT_URL,
+                        help="URL of the screenplay to download (default: Twelve Monkeys)")
+    
+    args = parser.parse_args()
+    script_url = args.url
+
+    # Extract the script name from the URL
+    script_name = os.path.basename(script_url).split(".")[0] + ".txt"
+    save_path = os.path.join(save_dir, script_name)
+
+    # Download the script
+    download_script(script_url, save_dir)

screenplays/send_screenplay.py

@@ -0,0 +1,48 @@
+import os
+import requests
+import psycopg2
+import math
+
+def chunk_text(text, chunk_size=4096):
+    words = text.split()
+    return [' '.join(words[i:i+chunk_size]) for i in range(0, len(words), chunk_size)]
+
+def is_screenplay_in_db(cursor, title):
+    cursor.execute("SELECT COUNT(*) FROM items WHERE title LIKE %s", (f"{title}%",))
+    return cursor.fetchone()[0] > 0
+
+def send_screenplay(file_path, cursor):
+    with open(file_path, 'r', encoding='utf-8') as file:
+        content = file.read()
+
+    title = os.path.splitext(os.path.basename(file_path))[0]
+    
+    if is_screenplay_in_db(cursor, title):
+        print(f"Screenplay '{title}' already exists in the database. Skipping.")
+        return
+
+    chunks = chunk_text(content)
+
+    for i, chunk in enumerate(chunks):
+        payload = {
+            "title": f"{title}_chunk_{i+1}",
+            "doc_text": chunk
+        }
+        response = requests.post("http://localhost:8080/add_document", json=payload)
+        print(f"Chunk {i+1} response: {response.status_code}")
+
+def process_scripts_folder():
+    conn = psycopg2.connect("dbname=ragtag user=jc password=!1newmedia host=localhost")
+    cursor = conn.cursor()
+
+    scripts_folder = "scripts"
+    for filename in os.listdir(scripts_folder):
+        if filename.endswith(".txt"):
+            file_path = os.path.join(scripts_folder, filename)
+            send_screenplay(file_path, cursor)
+
+    cursor.close()
+    conn.close()
+
+if __name__ == "__main__":
+    process_scripts_folder()