Update app.py

app.py

@@ -227,10 +227,120 @@ iface = gr.Interface(
     flagging_mode="never"
 )
 
+# The code remains the same as in the previous artifact, so I'll omit it here for brevity.
+# The changes will be in the tutorial_md variable.
+
 tutorial_md = """
 # Embedding Comparison Tool Tutorial
 
-This tool allows you to compare different embedding models and retrieval strategies for document search. Here's how to use it:
+This tool allows you to compare different embedding models and retrieval strategies for document search. Before we dive into how to use the tool, let's cover some important concepts.
+
+## What is RAG?
+
+RAG stands for Retrieval-Augmented Generation. It's a technique that combines the strength of large language models with the ability to access and use external knowledge. RAG is particularly useful for:
+
+- Providing up-to-date information
+- Answering questions based on specific documents or data sources
+- Reducing hallucinations in AI responses
+- Customizing AI outputs for specific domains or use cases
+
+RAG is good for applications where you need accurate, context-specific information retrieval combined with natural language generation. This includes chatbots, question-answering systems, and document analysis tools.
+
+## Key Components of RAG
+
+### 1. Document Loading
+
+This is the process of ingesting documents from various sources (PDFs, web pages, databases, etc.) into a format that can be processed by the RAG system. Efficient document loading is crucial for handling large volumes of data.
+
+### 2. Document Splitting
+
+Large documents are often split into smaller chunks for more efficient processing and retrieval. The choice of splitting method can significantly impact the quality of retrieval results.
+
+### 3. Vector Store and Embeddings
+
+Embeddings are dense vector representations of text that capture semantic meaning. A vector store is a database optimized for storing and querying these high-dimensional vectors. Together, they allow for efficient semantic search.
+
+### 4. Retrieval
+
+This is the process of finding the most relevant documents or chunks based on a query. The quality of retrieval directly impacts the final output of the RAG system.
+
+## Why is this important?
+
+Understanding and optimizing each component of the RAG pipeline is crucial because:
+
+1. It affects the accuracy and relevance of the information retrieved.
+2. It impacts the speed and efficiency of the system.
+3. It determines the scalability of your solution.
+4. It influences the overall quality of the generated responses.
+
+## Impact of Parameter Changes
+
+Changes in various parameters can have significant effects:
+
+- **Chunk Size**: Larger chunks provide more context but may reduce precision. Smaller chunks increase precision but may lose context.
+- **Overlap**: More overlap can help maintain context between chunks but increases computational load.
+- **Embedding Model**: Different models have varying performance across languages and domains.
+- **Vector Store**: Affects query speed and the types of searches you can perform.
+- **Retrieval Method**: Impacts the diversity and relevance of retrieved documents.
+
+## Detailed Parameter Explanations
+
+### Embedding Model
+
+The embedding model translates text into numerical vectors. The choice of model affects:
+
+- **Language Coverage**: Some models are monolingual, others are multilingual.
+- **Domain Specificity**: Models can be general or trained on specific domains (e.g., legal, medical).
+- **Vector Dimensions**: Higher dimensions can capture more information but require more storage and computation.
+
+#### Vocabulary Size
+
+The vocab size refers to the number of unique tokens the model recognizes. It's important because:
+
+- It affects the model's ability to handle rare words or specialized terminology.
+- Larger vocabs can lead to better performance but require more memory.
+- It impacts the model's performance across different languages (larger vocabs are often better for multilingual models).
+
+### Split Strategy
+
+- **Token**: Splits based on a fixed number of tokens. Good for maintaining consistent chunk sizes.
+- **Recursive**: Splits based on content, trying to maintain semantic coherence. Better for preserving context.
+
+### Vector Store Type
+
+- **FAISS**: Fast, memory-efficient. Good for large-scale similarity search.
+- **Chroma**: Offers additional features like metadata filtering. Good for more complex querying needs.
+
+### Search Type
+
+- **Similarity**: Returns the most similar documents. Fast and straightforward.
+- **MMR (Maximum Marginal Relevance)**: Balances relevance with diversity in results. Useful for getting a broader perspective.
+
+## MTEB (Massive Text Embedding Benchmark)
+
+MTEB is a comprehensive benchmark for evaluating text embedding models across a wide range of tasks and languages. It's useful for:
+
+- Comparing the performance of different embedding models.
+- Understanding how models perform on specific tasks (e.g., classification, clustering, retrieval).
+- Selecting the best model for your specific use case.
+
+### Finding Embeddings on MTEB Leaderboard
+
+To find suitable embeddings using the MTEB leaderboard (https://huggingface.co/spaces/mteb/leaderboard):
+
+1. Look at the "Avg" column for overall performance across all tasks.
+2. Check performance on specific task types relevant to your use case (e.g., Retrieval, Classification).
+3. Consider the model size and inference speed for your deployment constraints.
+4. Look at language-specific scores if you're working with non-English text.
+5. Click on model names to get more details and links to the model pages on Hugging Face.
+
+When selecting a model, balance performance with practical considerations like model size, inference speed, and specific task performance relevant to your application.
+
+By understanding these concepts and parameters, you can make informed decisions when using the Embedding Comparison Tool and optimize your RAG system for your specific needs.
+
+## Using the Embedding Comparison Tool
+
+Now that you understand the underlying concepts, here's how to use the tool:
 
 1. **File Upload**: Optionally upload a file (PDF, DOCX, or TXT) or leave it empty to use files in the `./files` directory.
 
@@ -264,6 +374,7 @@ You can download the results as CSV files for further analysis.
 Experiment with different settings to find the best combination for your specific use case!
 """
 
+# The rest of the code remains the same
 iface = gr.TabbedInterface(
     [iface, gr.Markdown(tutorial_md)],
     ["Embedding Comparison", "Tutorial"]