In-Depth Understanding of Vector Search for RAG and Generative AI Applications

You might have used large language models like GPT-3.5  GPT-4o  or any of the other models  Mistral or Perplexity  and these large language models are awe-inspiring with what they can do and how much of a grasp they have of language.   So  today I was chatting with an LLM  and I wanted to know about my companys policy if I work from India instead of the UK. You can see I got a really generic answer  and then it asked me to consult my company directly.   The second question I asked was  Who won the last T20 Worldcup and we all know that India won the ICC T20 2024 World Cup.   Theyre large language models; theyre very good at next-word predictions; theyve been trained on public knowledge up to a certain point; and theyre going to give us outdated information.   So  how can we incorporate domain knowledge into an LLM so that we can get it to answer those questions? There are three main ways that people will go about incorporating domain knowledge:   Prompt Engineering: In context learning  we can derive an LLM to solve by putting in a lot of effort using prompt engineering; however  it will never be able to answer if it has never seen that information.Fine Tuning: Learning new skills; in this case  you start with the base model and train it on the data or skill you want it to achieve. And it will be really expensive to train the model on your data.Retrieval Augmentation: Learning new facts temporarily to answer questionsHow do RAGs work?When I want to ask about any policy in my company  I will store it in a database and ask a question regarding the same. Our search system will search the document with the most relevant results and get back the information. We call this information knowledge. We will pass the knowledge and query to an LLM  and we will get the desired results.   We understand that if we provide LLM domain knowledge  then it will be able to answer perfectly. Now everything boils down to the retrieval part. Responses are only as good as retrieving data. So  lets understand how we can improve document retrieval.   How do we search?Traditional search has been keyword search-based  but then keyword search has this issue of the vocabulary gap. So  if I say Im looking for underwater activities but that word underwater is nowhere in our knowledge base at all  then a keyword search would never match scuba and snorkeling  so thats why we want to have a vector-based retrieval as well  which can find things by semantic similarity. A vector-based search is going to help you realise that scuba diving and snorkeling are semantically similar to underwater and be able to return those so thats why were talking about the importance of vector embedding today. So  lets go deep into vectors   Embeddings: The Back Bone of LLMsYoure not alone if the term embeddings has ever left you scratching your head or feeling lost in a sea of technicallevelup.gitconnected.com   Vector EmbeddingsVector Embeddings takes some input like a word or a sentence and then it sends it through through some embedding model. Then get back a list of floating point numbers and the amount of numbers is going to vary based on the actual model that youre using.   So  here I have a table of the most common models we see. We have word2vec and that only takes an input of a single word at a time and the resulting vectors have a length of 300. What weve seen in the last few years is models based off of LLMs and these can take into much larger inputs which is really helpful because then we can search on more than just words.   The one that many people use now is OpenAIs ada-002 which takes the text of up to 8 191 tokens and it produces vectors that are 1536. You need to be consistent with what model you use  so you do want to make sure that you are using the same model for indexing the data and for searching.   You can learn more about the basics of vector search in my previous blog.   import json import os import azure.identity import dotenv import numpy as np import openai import pandas as pd # Set up OpenAI client based on environment variables dotenv.load_dotenv() AZURE_OPENAI_SERVICE = os.getenv(AZURE_OPENAI_SERVICE) AZURE_OPENAI_ADA_DEPLOYMENT = os.getenv(AZURE_OPENAI_ADA_DEPLOYMENT) azure_credential = azure.identity.DefaultAzureCredential() token_provider = azure.identity.get_bearer_token_provider(azure_credential  https://cognitiveservices.azure.com/.default) openai_client = openai.AzureOpenAI( api_version=2023-07-01-preview  azure_endpoint=fhttps://{AZURE_OPENAI_SERVICE}.openai.azure.com  azure_ad_token_provider=token_provider)In the above code first  we will just set up a connection to OpenAI. Im using Azure.   def get_embedding(text): get_embeddings_response = openai_client.embeddings.create(model=AZURE_OPENAI_ADA_DEPLOYMENT  input=text) return get_embeddings_response.data[0].embedding def get_embeddings(sentences): embeddings_response = openai_client.embeddings.create(model=AZURE_OPENAI_ADA_DEPLOYMENT  input=sentences) return [embedding_object.embedding for embedding_object in embeddings_response.data]We have these functions here that are just wrappers for creating embeddings using the Ada 002 model   # optimal size to embed is ~512 tokens vector = get_embedding(A dog just walked past my house and yipped yipped like a Martian) # 8192 tokens limitWhen we vectorise the sentence  A dog just walked past my house and yipped yipped like a Martian  we can write a long sentence and we can calculate the embedding. No matter how long is the sentence  we will get the embeddings of the same length which is 1536.   When were indexing documents for RAG chat apps were often going to be calculating embeddings for entire paragraphs up to 512 tokens is best practice. You dont want to calculate the embedding for an entire book because thats above the limit of 8192 tokens but also because if you try to embed long text then the nuance is going to be lost when youre trying to compare one vector to another vector.   Vector SimilarityWe compute embeddings so that we can calculate the similarity between inputs. The most common distance measurement is cosine similarity.   We can use other methods to calculate the distance between the vectors as well; however  it is recommended to use cosine similarity when we are using the ada-002 embedding model and below is the formula to calculate the cosine similarities of 2 vectors.   def cosine_sim(a b): return dot(a b)/(mag(a) * mag(b))how you calculate cosine similarities its the dot product over the product of the magnitudes and it tells us how similar two vectors are. What is the angle between these two vectors in multi-dimensional space? so here we visualizing in two-dimensional space because we can not visualize 1536 dimensions   If the vectors are close then theres a very small Theta and that means you know your angle Theta is near zero which means the cosine of the angle is near 1. As the vectors get farther and further away then your cosine goes down to zero and potentially even to negative 1   def cosine_similarity(a  b): return np.dot(a  b) / (np.linalg.norm(a) * np.linalg.norm(b)) sentences1 = ['The new movie is awesome'  'The new movie is awesome'  'The new movie is awesome'] sentences2 = ['djkshsjdkhfsjdfkhsd'  'This recent movie is so good'  'The new movie is awesome'] embeddings1 = get_embeddings(sentences1) embeddings2 = get_embeddings(sentences2) for i in range(len(sentences1)): print(f{sentences1[i]} \t\t {sentences2[i]} \t\t Score: {cosine_similarity(embeddings1[i]  embeddings2[i]):.4f})So here Ive got a function to calculate the cosine similarity and Im using numpy to do the math for me since thatll be nice and efficient and now Ive got three sentences that are all the same and then these sentences which are different and Im going to get the embeddings for each of these sets of sentences and then just compare them to each other.   What we see is that you know when the two sentences are the same then we see a cosine similarity of one thats what we expect and then when a sentence is very similar  then we see a cosine similarity of 0.91 for sentence 2 and then sentence 1 is 0.74.   Now when you look at this its hard to think about whether the 0.75 means this is pretty similar or does it mean its pretty dissimilar   When you do similarity with the Ada 002 model  theres generally a very tight range between about .65 and 1(speaking from my experience and what I have seen so far) so this .75 is dissimilar.   Vector SearchNow the next step is to be able to do a vector search because everything we just did above was for similarity within the existing data set. What we want to be able to do is be able to search user queries.   We will compute the embedding vector for that query using the same model that we did our embeddings with for the knowledge base and then we look in our Vector database and find the K closest vectors for that user query vector   # Load in vectors for movie titles with open('openai_movies.json') as json_file: movie_vectors = json.load(json_file)# Compute vector for query query = My Neighbor Totoro embeddings_response = openai_client.embeddings.create(model=AZURE_OPENAI_ADA_DEPLOYMENT  input=[query]) vector = embeddings_response.data[0].embedding # Compute cosine similarity between query and each movie title scores = [] for movie in movie_vectors: scores.append((movie  cosine_similarity(vector  movie_vectors[movie]))) # Display the top 10 results df = pd.DataFrame(scores  columns=['Movie'  'Score']) df = df.sort_values('Score'  ascending=False) df.head(10)Ive got my query which is My Neighbor Totoro because those movies were only Disney movies and as far as I know My Neighbor Totoro is not a Disney were going to do a comprehensive search here so for every single movie in those vectors were going to calculate the cosine similarity between the query vector and the vector for that movie and then were going to create a data frame and sort it so that we can see the most similar ones.   Vector DatabaseWe have learned how to use vector search. So moving on  how do we store our vectors? We want to store in some sort of database usually a vector database or a database that has a vector extension. We need something that can store vectors and ideally knows how to index vectors.   Navigating the World of Vector Databases: Understanding Their Concepts  Applications  and ExamplesHello! Lets explain Vector Databases with an example. Imagine a vast library filled with books  each volumetalibilat.medium.com   Below is a little example of postgress code using the PG Vector extension:   CREATE EXTENSION vector; CREATE TABLE items (id bigserial PRIMARY KEY  embedding vector(1536)); INSERT INTO items (embedding) VALUES ('[0.0014701404143124819  0.0034404152538627386  -0.01280598994344729 ...]'); CREATE INDEX ON items USING hnsw (embedding vector_cosine_ops); SELECT * FROM items ORDER BY embedding <=> '[-0.01266181  -0.0279284 ...]' LIMIT 5;Here we declare our Vector column and we say its going to be a vector with 1536 dimensions and then we can insert our vectors in there and then we could do a select where were checking to see which embedding is closest to the embedding that were interested. This is an index using hnsw which is an approximation algorithm.   On Azure  we have several options for Vector databases. We do have Vector support in the MongoDB vcore and also in the cosmos DB for postgress. Thats a way you could keep your data where it is  for example; if youre making a RAG chat application on your product inventory and your product inventory changes all the time and its already in the cosmos DB. Then it makes sense to take advantage of the vector capabilities there.   Otherwise  we have Azure AI search  a dedicated search technology that does not just do vector search but also keyword search. It has a lot more features. It can index things from many sources and this is what I generally recommend for a really good search quality.   Im going to use Azure AI Search for the rest of this blog and were going to talk about all its features how it integrates and what makes it a really good retrieval system.   Azure AI SearchAzure AI Search is a search-as-a-service in the cloud  providing a rich search experience that is easy to integrate into custom applications  and easy to maintain because all infrastructure and administration is handled for you.   AI search has vector search which you can use via your Python SDK which Im going to use in the blog below  but also with semantic kernel LangChain  LlamaIndex or any of those packages that youre using  most of them do have a support for AI search as the RAG knowledge base.   To use AI Search first we will import the libraries.   import os import azure.identity import dotenv import openai from azure.search.documents import SearchClient from azure.search.documents.indexes import SearchIndexClient from azure.search.documents.indexes.models import ( HnswAlgorithmConfiguration  HnswParameters  SearchField  SearchFieldDataType  SearchIndex  SimpleField  VectorSearch  VectorSearchAlgorithmKind  VectorSearchProfile  ) from azure.search.documents.models import VectorizedQuery dotenv.load_dotenv()Initialize Azure search variables   # Initialize Azure search variables AZURE_SEARCH_SERVICE = os.getenv(AZURE_SEARCH_SERVICE) AZURE_SEARCH_ENDPOINT = fhttps://{AZURE_SEARCH_SERVICE}.search.windows.netSet up OpenAI client based on environment variables   # Set up OpenAI client based on environment variables dotenv.load_dotenv() AZURE_OPENAI_SERVICE = os.getenv(AZURE_OPENAI_SERVICE) AZURE_OPENAI_ADA_DEPLOYMENT = os.getenv(AZURE_OPENAI_ADA_DEPLOYMENT) azure_credential = azure.identity.DefaultAzureCredential() token_provider = azure.identity.get_bearer_token_provider(azure_credential  https://cognitiveservices.azure.com/.default) openai_client = openai.AzureOpenAI( api_version=2023-07-01-preview  azure_endpoint=fhttps://{AZURE_OPENAI_SERVICE}.openai.azure.com  azure_ad_token_provider=token_provider)Defining a function to get the embeddings.   def get_embedding(text): get_embeddings_response = openai_client.embeddings.create(model=AZURE_OPENAI_ADA_DEPLOYMENT  input=text) return get_embeddings_response.data[0].embeddingCreating a Vector IndexNow we can create an index  we will name it index-v1. It has a couple of fields   ID field: thats like our primary keyEmbedding field: That is going to be a vector and we tell it how many dimensions its going to have. Then we also give it a profile embedding_profile.AZURE_SEARCH_TINY_INDEX = index-v1 index = SearchIndex( name=AZURE_SEARCH_TINY_INDEX  fields=[ SimpleField(name=id  type=SearchFieldDataType.String  key=True)  SearchField(name=embedding  type=SearchFieldDataType.Collection(SearchFieldDataType.Single)  searchable=True  vector_search_dimensions=3  vector_search_profile_name=embedding_profile) ]  vector_search=VectorSearch( algorithms=[HnswAlgorithmConfiguration( # Hierachical Navigable Small World  IVF name=hnsw_config  kind=VectorSearchAlgorithmKind.HNSW  parameters=HnswParameters(metric=cosine)  )]  profiles=[VectorSearchProfile(name=embedding_profile  algorithm_configuration_name=hnsw_config)] ) ) index_client = SearchIndexClient(endpoint=AZURE_SEARCH_ENDPOINT  credential=azure_credential) index_client.create_index(index)In VecrotSearch() we will describe which algorithm or indexing strategy we want to use and were going to use hnsw which stands for hierarchical navigable small world. Theres are a couple other options like IVF  Exhaustive KNN and some others.   AI search supports hnsw because it works well and theyre able to do it efficiently at scale. So  were going to say its hnsw and we can tell it like what metric to use for the similarity calculations we can also customize other hnsw parameters if youre familiar with that.   azure.search.documents.indexes.models.HnswParameters classContains the parameters specific to the HNSW algorithm.learn.microsoft.com   Search using vector similarityOnce the vector is created the index and now we just are going to upload documents   search_client = SearchClient(AZURE_SEARCH_ENDPOINT  AZURE_SEARCH_TINY_INDEX  credential=azure_credential) search_client.upload_documents(documents=[ {id: 1  embedding: [1  2  3]}  {id: 2  embedding: [1  1  3]}  {id: 3  embedding: [4  5  6]}])Search using vector similarityNow will search through the documents. Were not doing any sort of text search  were only doing a vector query search.   r = search_client.search(search_text=None  vector_queries=[ VectorizedQuery(vector=[-2  -1  -1]  k_nearest_neighbors=3  fields=embedding)]) for doc in r: print(fid: {doc['id']}  score: {doc['@search.score']})Were asking for the 3 nearest neighbors and were telling it to search the embedding_field because you could have multiple Vector Fields.   We do this search and we can see the output scores. The score in this case is not necessarily the cosine similarity because the score can consider other things as well and theres some documentation about what score means in different situations   Vector relevance and ranking - Azure AI SearchExplains the concepts behind vector relevance  scoring  including how matches are found in vector space and ranked inlearn.microsoft.com   r = search_client.search(search_text=None  vector_queries=[ VectorizedQuery(vector=[-2  -1  -1]  k_nearest_neighbors=3  fields=embedding)]) for doc in r: print(fid: {doc['id']}  score: {doc['@search.score']})We see much lower scores if we put vector = [-2  -1  -1]. I usually dont look at the absolute scores myself you can but I typically look at the relative scores   Searching on Large IndexAZURE_SEARCH_FULL_INDEX = large-index search_client = SearchClient(AZURE_SEARCH_ENDPOINT  AZURE_SEARCH_FULL_INDEX  credential=azure_credential) search_query = learning about underwater activities search_vector = get_embedding(search_query) r = search_client.search(search_text=None  top=5  vector_queries=[ VectorizedQuery(vector=search_vector  k_nearest_neighbors=5  fields=embedding)]) for doc in r: content = doc[content].replace(\n   )[:150] print(fScore: {doc['@search.score']:.5f}\tContent:{content})Vector search strategiesDuring vector query execution  the search engine searches for similar vectors to determine which candidates to return in search results. Depending on how you indexed the vector information  the search for suitable matches can be extensive or limited to near neighbours to speed up processing. Once candidates have been identified  similarity criteria are utilised to rank each result based on the strength of the match.   There are 2 famous vector search algorithms in Azure:   Exhaustive KNN: runs a brute-force search across the whole vector space.HNSW runs an approximate nearest neighbour (ANN) search.Only vector fields labelled as searchable in the index or searchFields in the query are used for searching and scoring.   When to use exhaustive KNN?Exhaustive KNN computes the distances between all pairs of data points and identifies the precise k nearest neighbours for a query point. It is designed for cases in which strong recall matters most and users are ready to tolerate the trade-offs in query latency. Because exhaustive KNN is computationally demanding  it should be used with small to medium datasets or when precision requirements outweigh query efficiency considerations.   r = search_client.search( None  top = 5  vector_queries = [VectorizedQuery( vector = search_vector  k_nearest_neighbour = 5  field = embedding)])A secondary use case is to create a dataset to test the approximate closest neighbour algorithms recall. Exhaustive KNN can be used to generate a ground truth collection of nearest neighbours.   When to use HNSW?During indexing  HNSW generates additional data structures to facilitate speedier search  arranging data points into a hierarchical graph structure. HNSW includes various configuration options that can be adjusted to meet your search applications throughput  latency  and recall requirements. For example  at query time  you can specify options for exhaustive search  even if the vector field is HNSW-indexed.   r = search_client.search( None  top = 5  vector_queries = [VectorizedQuery( vector = search_vector  k_nearest_neighbour = 5  field = embedding  exhaustive = True)])During query execution  HNSW provides quick neighbour queries by traversing the graph. This method strikes a balance between search precision and computing efficiency. HNSW is suggested for most circumstances because of its efficiency when searching massive data sets.   Filtered vector searchNow we have other capabilities when were doing Vector queries. You can set vector filter modes on a vector query to specify whether you want to filter before or after query execution.   Filters determine the scope of a vector query. Filters are set on and iterate over nonvector string and numeric fields attributed as filterable in the index  but the purpose of a filter determines what the vector query executes over: the entire searchable space  or the contents of a search result.   With a vector query now one thing you have to keep in mind is whether you should be doing a pre-filter or post-filter  you generally want to do a pre-filter and this means that youre first doing this filter and then doing the vector search and the reason you want this because  think about  if you did a post filter then there are some chances that you might not find a relevant vector match after that which will return empty results. Instead  what you want to do is filter all the documents and then query the vectors.   r = search_client.search( None  top = 5  vector_queries = [VectorizedQuery( vector = query_vector  k_nearest_neighbour = 5  field = embedding )] vector_filter_mode = VectorFilterMode.PRE_FILTER  filter = your filter here )Multi-vector searchWe also get support for multi-vector scenarios so for example if you have an embedding for the title of a document that was different from the embedding for the body of the document. You can search these separately you could you know search these at the same time with different.   We use this a lot if were doing multimodal queries so if we have both an image embedding and a text embedding we might want to search both of those embeddings   Azure AI search not only supports text search but also image and audio search as well. Lets see an example of an image search.   import os import dotenv from azure.identity import DefaultAzureCredential  get_bearer_token_provider from azure.search.documents import SearchClient from azure.search.documents.indexes import SearchIndexClient from azure.search.documents.indexes.models import ( HnswAlgorithmConfiguration  HnswParameters  SearchField  SearchFieldDataType  SearchIndex  SimpleField  VectorSearch  VectorSearchAlgorithmKind  VectorSearchProfile  ) from azure.search.documents.models import VectorizedQuery dotenv.load_dotenv() AZURE_SEARCH_SERVICE = os.getenv(AZURE_SEARCH_SERVICE) AZURE_SEARCH_ENDPOINT = fhttps://{AZURE_SEARCH_SERVICE}.search.windows.net AZURE_SEARCH_IMAGES_INDEX = images-index4 azure_credential = DefaultAzureCredential(exclude_shared_token_cache_credential=True) search_client = SearchClient(AZURE_SEARCH_ENDPOINT  AZURE_SEARCH_IMAGES_INDEX  credential=azure_credential)Creating a Search Index for ImagesWe create a search index for images so this one has ID = file name and embedding this time the vector search dimensions is 1024 because that is the dimensions of the embeddings that come from the computer vision model so its slightly different length than the ada-002 and everything else is the same.   index = SearchIndex( name=AZURE_SEARCH_IMAGES_INDEX  fields=[ SimpleField(name=id  type=SearchFieldDataType.String  key=True)  SimpleField(name=filename  type=SearchFieldDataType.String)  SearchField(name=embedding  type=SearchFieldDataType.Collection(SearchFieldDataType.Single)  searchable=True  vector_search_dimensions=1024  vector_search_profile_name=embedding_profile) ]  vector_search=VectorSearch( algorithms=[HnswAlgorithmConfiguration( name=hnsw_config  kind=VectorSearchAlgorithmKind.HNSW  parameters=HnswParameters(metric=cosine)  )]  profiles=[VectorSearchProfile(name=embedding_profile  algorithm_configuration_name=hnsw_config)] ) ) index_client = SearchIndexClient(endpoint=AZURE_SEARCH_ENDPOINT  credential=azure_credential) index_client.create_index(index)Configure Azure Computer Vision multi-modal embeddings APIHere we are integrating with the Azure Computer Vision service to obtain embeddings for images and text. It uses a bearer token for authentication  retrieves model parameters for the latest version  and defines functions to get the embeddings. The `get_image_embedding` function reads an image file  determines its MIME type  and sends a POST request to the Azure service  handling errors by printing the status code and response if it fails. Similarly  the `get_text_embedding` function sends a text string to the service to retrieve its vector representation. Both functions return the resulting vector embeddings.   import mimetypes import os import requests from PIL import Image token_provider = get_bearer_token_provider(azure_credential  https://cognitiveservices.azure.com/.default) AZURE_COMPUTERVISION_SERVICE = os.getenv(AZURE_COMPUTERVISION_SERVICE) AZURE_COMPUTER_VISION_URL = fhttps://{AZURE_COMPUTERVISION_SERVICE}.cognitiveservices.azure.com/computervision/retrieval def get_model_params(): return {api-version: 2023-02-01-preview  modelVersion: latest} def get_auth_headers(): return {Authorization: Bearer  + token_provider()} def get_image_embedding(image_file): mimetype = mimetypes.guess_type(image_file)[0] url = f{AZURE_COMPUTER_VISION_URL}:vectorizeImage headers = get_auth_headers() headers[Content-Type] = mimetype # add error checking response = requests.post(url  headers=headers  params=get_model_params()  data=open(image_file  rb)) if response.status_code != 200: print(image_file  response.status_code  response.json()) return response.json()[vector] def get_text_embedding(text): url = f{AZURE_COMPUTER_VISION_URL}:vectorizeText return requests.post(url  headers=get_auth_headers()  params=get_model_params()  json={text: text}).json()[vector]Add image vector to search indexNow we process each image file in the product_images directory. For each image  it calls the get_image_embedding function to get the image's vector representation (embedding). Then  it uploads this embedding to a search client along with the image's filename and a unique identifier (derived from the filename without its extension). This allows the images to be indexed and searched based on their content.   for image_file in os.listdir(product_images): image_embedding = get_image_embedding(fproduct_images/{image_file}) search_client.upload_documents(documents=[{ id: image_file.split(.)[0]  filename: image_file  embedding: image_embedding}])Query Using an Imagequery_image = query_images/tealightsand_side.jpg Image.open(query_image)query_vector = get_image_embedding(query_image) r = search_client.search(None  vector_queries=[ VectorizedQuery(vector=query_vector  k_nearest_neighbors=3  fields=embedding)]) all = [doc[filename] for doc in r] for filename in all: print(filename)Now  we are getting the embedding for a query image and searching for the top 3 most similar image embeddings using a search client. It then prints the filenames of the matching images.   Image.open(product_images/ + all[0])Now lets take it to the next level and search images using text.   query_vector = get_text_embedding(lion king) r = search_client.search(None  vector_queries=[ VectorizedQuery(vector=query_vector  k_nearest_neighbors=3  fields=embedding)]) all = [doc[filename] for doc in r] for filename in all: print(filename) Image.open(product_images/ + all[0])If you see here  we searched for Lion King and not only did it get the reference of Lion King but also was able to read the texts on images and bring back the best match from the dataset.   I hope you enjoyed reading the blog and learned something new. In the upcoming blogs  I will be talking more about Azure AI Search.   Thank you for reading!   Lets connect on LinkedIn!   GitHub   You might be interested in Reading!Choosing the Right Generative AI Framework-LangChain  LlamaIndex  Haystack  or Hugging FaceSummarising Large Documents with GPT-4oHow does LlamaIndex compare to LangChain in terms of ease of use for beginners?Pre-training vs. Fine-tuning [With code implementation]Costs of Hosting Open Source LLMs vs Closed Sourced (OpenAI)Embeddings: The Back Bone of LLMsHow to Use a Fine-Tuned Language Model for Summarization


Can You Actually Beat the Dealer in Blackjack?  Simulation of Most Popular Strategies

In this article I explore if it is actually possible to beat the blackjack dealer using strategic thought. Of course  the underlying idea here is to show the use of simulation and how a game can be modeled mathematically. (But please do feel to try out any of below mentioned strategies if interested!)   The framework discussed below implements a blackjack game simulation and compares the performance of a few selected counting strategies.   Knowledgeable Blackjack players are able to beat the casinos using a technique called card counting. Card counting is technically legal  however a casino reserves the right to deny entry to any individual they suspect of using this skill to their advantage.   A statistical analysis is done on these strategies simulating multiple independent blackjack games. Blackjack  as most games in Casino  is heavily dependent on luck. Without counting cards or using any strategy  it is almost always the dealer that gets an advantage.   While  per the rules and design of the game  chances of winning are less than 50%  just following the well-known Basic Strategy for counting cards can decrease the advantage of the house and increase the probability of winning to ~50%. There are various Blackjack strategies developed and taught all over the world  this analysis builds a system to compare these strategies and determine which is the most profitable approach. Multiple independent blackjack games are thus simulated to understand the profit and loss in various scenarios. On comparison  Hi-Opt I and Hi-Opt II have been identified to maximise win expectation.   Rules of the Game: Blackjack allows for multiple players. Each players plays the Casino dealer  and each game is independent. Each game may contain multiple rolls. In a roll  using the cards  both the player and the dealer try to generate highest number possible without crossing the sum of 21. Crossing this threshold indicates going bust  meaning you have lost. As the player goes first  there is always a slight advantage to the dealer. Please find below  a summarized view of common scenarios during the game.   The players cards are dealt face up. One of the dealers cards is dealt face down while the other is dealt face up.Each card from 2 to 10 has a value corresponding to its number.The face cards are worth 10An ace is worth either 1 or 11  depending on what is advantageous during the game.A player can either decide to Hit that is request for another card or Stand that is keep current cards during a game.A player can also Double the amount of bet made during the game and receives just one additional card.If the player is dealt a pair of same cards  they can Split their hand. That is  they can play two separate games using their cards.While exploring these strategies  this analysis explores two major areas    Identifying point of convergence for expected wins in independent blackjack games (After how many simulations can winning probability be determined)Explore and compare different card counting strategiesIn the upcoming sections  scope and assumptions are defined  followed by brief explanation of the development architecture  describing the logic used to simulate the game of blackjack with different strategies. Key findings are then described that set forth the results of the analysis on Black Jack simulation strategies.   Key Assumptions:I focus on just one player and the dealer. To reduce complexity  options like splitting  doubling down  surrendering and insurance are not considered. A six-deck card system has been considered for this simulation. A player is awarded 1 point on each win  however in case of Blackjack the reward is 3/2 points.   Ten counting strategies are considered for this analysis. Please find below the values associated with each card for different strategies. In case of No Strategy  no counting strategy is followed  apart from basic Blackjack Strategy.   Development and Architecture:Implementation is primarily done in Python using Jupyter Notebook (Fig. 1). Card values for each card of the deck is taken as input for each strategy using a CSV file. The Blackjack simulation script inputs this information along with number of simulations to be run. Multiple blackjack games are then simulated and expected winning is logged.   Script Overview:   Counting strategies CSV and number of simulations to be run are input by the user.Cards are dealt to the player and the dealer.Running count and True count (based on values defined in strategies.csv) are maintained based on open card count throughout the game.In case no blackjack is achieved by either player or dealer both have an option to Stay or Hit.Player choice is determined by threshold on true count of currently open cards.A game ends if < 12 cards are left in the deck or < 1 card is left in the last roll.Winnings/Losses are determined at end of each roll and count is logged.2000 blackjack games are played using each strategy.Full code is available here: https://github.com/eramkhan94/blackjack_simulator   Key FindingsIdentifying point of convergence: Blackjack games were simulated 10 000 times with No Strategy. Expected winnings were logged and plotted (Fig. 2). The simulation reached convergence with a predefined delta of 0.01  between the average expected win of previous 200 iterations (i.e. 12001400) and next 200 iterations (i.e. 14001600)  reaching ~1600 games. Additional 400 iterations were added as a buffer to account for variance in convergence of different strategies. Thus  for further analysis and comparison  each strategy was simulated 2000 times and expected winnings were observed (Fig. 2).2. Strategy Analysis and Comparison: To further compare strategies  2000 games were simulated for each counting strategy 30 times and results were logged. The expected winnings for each strategy were compared against No Strategy using a one way ANOVA test. For six strategies out of ten  a p-value of less than 0.05 was observed indicating that the mean of expected winnings was significantly different than the expected winnings mean when No Strategy was followed (Fig. 4).   On further analysing the distribution of expected wins (Fig. 5)  it is found that Hi-Opt I has the highest expected wins at 50.43  followed by Hi-Opt II (50.31)  Halves (50.31)  Wizard Ace (50.30)  Red 7 (50.297) and Zen (50.291). Other strategies did not yield significantly different results than No Strategy. However all strategies resulted in higher expected wins compared to No Strategy. Highest variance in result was observed for Hi-Opt I  Halves and Omega II.   Conclusion:Counting strategies do offer an edge compared to an intuitive game in Blackjack. Hi-Opt I has resulted in maximum expected gain. In further analysis assumptions on player moves can be relaxed. The optimisation for player moves like splitting  doubling down will further generalise this framework and aid in developing and testing new blackjack strategies. Relaxing these assumptions may also result in an increased difference in wins when using a counting strategy and some counting strategies might yield higher expected wins.


Significance of Image Labeling in AI

The capability of AI to see and perceive its surroundings has myriad advantages. In this blog  we will explore further the invaluable role image labeling plays in training AI to see like humans.   Image labeling plays an invaluable role in AI by training machine learning models to identify an image and classes of objects within it. It plays a significant role across diverse industries by assisting organizations in decision-making.   It also enhances the accuracy and efficacy of AI algorithms. It helps in training machine learning models by extracting key information for computer vision models regarding the objects present in an image.   Image labeling is undoubtedly the driving force behind advanced technologies  including robotics  autonomous vehicles  medical imaging  and more. All these technologies become alive through image labeling.   Lets dive into the blog below to understand the key aspects of image labeling.   Image labeling involves the identification and marking of raw data like images  videos  texts  and more for training machine learning models. It helps in adding informative and meaningful labels to images to add context and aid machine learning models to learn from it.   Image labeling plays two critical roles in AI:   Develop working AI models: Tools and techniques in image labeling assist with highlighting or capturing key objects within an image. The labels aid in making images readable to machines. The highlighted images are used as training datasets for AI and machine learning models.Enhance computer vision: Image captions and annotations enhance accuracy through object detection. AI models can identify patterns by training AI and machine learning with labels.Techniques in Image LabelingImages need to be labeled accurately for training neural networks. There are three main techniques in image labeling:   Manual image labeling   This method requires manually defining labels for the whole image by drawing regions within an image and text descriptions for each area. This technique requires a human labeler to examine the image carefully  identify the objects  draw bounding boxes or polygons around the objects  and assign labels to every object. However  this technique suffers from two key limitations: labeling inconsistency and scalability.   Semi-automated image labeling   This technique of image labeling aids manual labelers by detecting the boundaries of objects within an image by offering a starting point to them. Image annotation software saves human labelers precious time by providing a partial map of objects in the image. This technique is useful when large datasets are involved as it hastens the labeling process without affecting accuracy.   Types of Image LabelingThere are eight types of image labeling as outlined below:   Image Classification   Image classification algorithms acquire images as input and automatically classify them into one of many labels or classes. A training dataset for image classification involves manually reviewing images and annotating them using labels via the algorithm.   Semantic Segmentation   This technique is used in computer vision for segmenting images. An image dataset is semantically segmented for locating all categories and classes.   Object Detection   An algorithm is used to detect an image within an image along with its location within an image frame. The area is indicated using various shapes  such as facial recognition dots used in facial recognition systems.   Skeletal Annotation   This technique is used to highlight body movement and alignment. Annotators use this technique for connecting lines on the human body. Dots are used to connect them at points of articulation.   2D Bounding Boxes   Through graphical representations  boundaries of objects are defined in a two-dimensional space. These boxes are used in computer vision and machine learning applications to segregate areas of interest for objects.   Key Point Annotation   This annotation technique is used for recognizing facial gestures  human poses  expressions  emotions  body language  and sentiments through connection of multiple dots.   Polygon Annotation   This technique involves marking and drawing shapes on a digital image as per their position and orientation. It also involves labeling images of irregular dimensions.   3D Cuboid Annotation   This technique involves the detection and recognition of 3D objects in images. It assists machines in estimating the depth of objects like vehicles  people  buildings  and other objects.   Use cases of Image LabelingImage labeling helps optimize real-life operations by training computers to interpret and comprehend the visual world the way humans do.   Retail   Image labeling using the 2D bounding box technique is used for labeling images in retail stores  including shirts  trousers  jackets  persons  etc. It helps in training machine learning models on diverse features  including price  color  design  etc.   Healthcare   Human organs in X-rays are labeled using the polygon technique. Machine learning models acquire training to identify deformities in human X-rays. Image labeling revolutionizes healthcare by spotting diseases  reducing costs  and enhancing patient experience.   Self-Driving or Autonomous Vehicles   Several car makers are adopting this technology  which depends on Semantic segmentation to label every pixel of an image. It helps identify roads  cars  traffic lights  poles  pedestrians  etc. It also helps make vehicles aware of their surroundings and sense obstacles in their path.   Emotion Detection   Human emotions or sentiments are detected using landmark annotation. This measures a persons emotional state in a given piece of content. It helps interpret product reviews  service reviews  movie reviews  email complaints/feedback  customer calls  meetings  and more.   Supply Chain   The lines and splines technique is used to label lanes within warehouses. This helps identify tracks according to their delivery location. It also assists robots in optimizing their path and automating the delivery chain  reducing human intervention and errors.   Image Labeling Services OutsourcingThe success of any AI and ML model depends on qualitative and accurate training datasets. Outsourcing image labeling services is an economical and efficient way for companies to handle their data training requirements.   Each image is labeled precisely to help ML algorithms detect and identify objects readily. Image labeling services assist in offering original data for building and optimizing AI models. By properly selecting the right image labeling service provider  businesses can reap the rewards of computer vision and AI-based solutions.   Key Benefits of Outsourcing Image Labeling ServicesAdvancements in AI and ML for positive results   Image labeling service providers specialize in labeling practices  so they are abreast of advancements in AI and ML models. They offer high-quality labeled images to ensure the AI model delivers accurate results. Better labeling enhances the AI models precision  resulting in positive ML results.   Unique and customized solutions for quality product development   The exposure to various business use cases helps in delivering unique and personalized solutions that cater to any AI need.   Automation and scalability for efficient business operations   Image labeling service providers offer an automated approach by minimizing the use of rulers and inspecting them. It helps in saving time and costs. Outsourcing helps in scaling without consuming the companys local resources.   Competitive advantage   AI assists companies in gaining an edge by enhancing their position among competitors. Labeled images help in deriving better data insights  which in turn results in strategizing.   ConclusionOutsourcing image labeling services is a viable option for businesses today as it helps them enhance their operational efficiency and expand their reach to various applications  such as autonomous vehicles and medical imaging.   The labeling of images and videos has enabled businesses to perform real-time analysis. What remains to be done is to allow machines to imagine and understand problems to be solved  along with a partner like machine learning to guide businesses through this intricate lifecycle.


Transformers & DSPy: The Perfect Combo to Start with LLMs

IntroductionWho has never used ChatGPT? Probably every single one of us! However  we do not face one of the latest and most promising developments in artificial intelligence only when we use ChatGPT. Large Language Models (LLMs) have been implemented across different companies from different domains and we are likely exposed to them every day.   For example  customer service teams use this technology to quickly handle basic queries and let agents focus on more demanding issues. Marketing agencies use it to support their creative side when building campaigns or to understand customer sentiment in social media posts. Or  Spotify could have used this technology to create the lyrics through audio transcription.   With so many possible use cases and the level of exposure that we have  this article aims to provide a simple but detailed explanation of how the backbone architecture of LLMs works and what novel concepts companies like Meta  Mistral AI and Google introduced to this architecture with their own models  LLaMA  Mixtral and Gemma.   Finally  we provide a practical implementation in python using the library DSPy of these LLMs to tackle different use cases such as sentiment analysis  summarization  and RAG systems.   As always  the code is available on Github.   Transformers: from tokenization to text generationFirst things first: TokenizationTokenization is the first step in the process of text generation. It is not part of the Transformer architecture but it is crucial to transform the raw input text into a suitable format  tokens  so that Transformers can process text.   SentencePiece [1] developed by Google  is one of the most used tokenizers for LLMs. The way it works is the following:   1. Splits the words into individual characters Imagine that the training data contains the following set of words and the frequency of each word has been determined  for example  word hug appears 10 times  pug 5 times  pun 12 times  and so on.   Based on this frequency  the words need to be split into individual characters as shown below:   2. Iteratively merges the most frequent character pairs into subwords until a predefined vocabulary size is reached From the example above  we can see that u followed by g appears 20 times ( hug 10 times + pug 5 times + hugs 5 times)  which is the most frequent symbol pair  therefore we merge them:   We repeat this step until it reaches the predefined vocabulary size which  for example  in our case is 9 tokens:   3. Once the vocabulary size is reached  we are ready to tokenize new data With a trained Transformer on the vocabulary created previously  it is time to generate text based on a new input.   Imagine that the word bug is part of the input text  then it will be tokenized as [b  ug] because the whole word bug is not present in the vocabulary but the subwords b and ug are.   However  if the input data has a word like mug since  m is not part of the vocabulary  then it will be encoded as [<unk>  ug].   4. Encoding Just like every machine learning model  Transformers do not handle textual data  therefore  the vocabulary is encoded through a token ID. For example  the token hug becomes the ID 1  p becomes the ID 2  ug becomes the ID 3 and so on and so forth.   And that is how this tokenizer works!   Transformer: How does it work?The Transformer architecture [2] was developed in 2017 to perform language translations. It can be split into 5 different components: word embeddings  positional embeddings  encoder  decoder and next word prediction  as shown in Figure 2. Each of these components will be explained in detail in the following subsections.   1. Word Embedding The first component in a Transformer is the word embedding. This component is responsible for converting each token of the input text into a d-dimensional vector  but how?   Lets consider the example in Figure 3  where we want to translate the sentence Large Language Models from English to Portuguese. After tokenization  the sentence is converted into three different tokens each one representing a word from the input text.   These tokens go through a linear layer  in our example with 4 nodes  that converts the tokens into a 4-dimensional vector that will be consumed by the remaining components of the architecture to predict the next work in the sequence. During the training phase  the weights of this layer are optimised through backpropagation in order to improve the vector representation of a token which  consequently  will improve how well the model is able to predict the next word in the sequence.   2. Positional Embedding After the vector representation has been generated  the second component  Positional Embedding  kicks in.   Since the Encoder and Decoder are order invariant which means they cannot distinguish the connotation of two sentences with the same words but different meanings  for example LLaMA is better than Mistral and Mistral is better than LLaMA  Positional Embedding is used to solve this issue and add a sense of order to the model.   The Positional Embedding consists in a new vector to encode positions based on sinusoidal functions (sine and cosine) and the token position. It also depends on the model dimension since the positional vector is added to the word embedding  therefore  it has to have the same dimension of the word embedding.   Considering the previous example  the positional vector to be added to the word embedding needs to have a dimension of 4. For that  four sinusoidal functions are considered and each one of them will return a value based on the token position to generate the positional vector as shown in Figure 4. This vector is added to the word embedding before feeding the Encoder.   3. Encoder and Self-Attention Self-Attention is the key part of the Transformer architecture since it uses the similarity between words in a sequence in order to encode the current word in the sequence. It is based on three main components: Query  Key and Value.   3.1 Query Queries are used to represent each word in a sequence through a new d-dimensional vector. This vector is created by feeding the output of the word and positional embedding through a layer with multiple nodes  in our case 4 nodes  as shown in Figure 5. Just like in the word embedding layer  the weights linked to each node are updated through backpropagation in order to improve next word prediction.   3.2 Key Keys follow the exact same logic as Queries  where there is a layer that receives the combined output of word and positional embedding to generate a new d-dimensional vector  as shown in Figure 6. Once again the weights of this layer are updated through backpropagation to improve model performance.   3.3 Value Just like before  another independent layer from the ones used in Queries and Keys  is used to generate a new d-dimensional vector based on the combined output of word and positional embedding to represent the Value of a token  as shown in Figure 7.   3.4 Combining Query  Key and Value Once all the three main components were calculated  it is time to combine all of them to generate the encoded vector of a token. Continuing with the same sentence as before  Large Language Models  the first step is to calculate the similarity between tokens.   For that  we calculate the dot product between the Query of the token that is being processed and the Key of the remaining tokens in the sequence and its own Key. Based on the output of the dot product  a.k.a. similarity  a Softmax function is applied to return the weight that each token will have to create the encoded vector of the token Large  as shown in Figure 8.   These weights are going to be multiplied with the Values of each token to generate the contextual vector based on the surrounded tokens for the token Large.   Finally  this contextual vector is summed to the combined output of the word and positional embedding through a skip connection before feeding a Feed Forward Neural Network (FNN) that generates the final encoded vector.   4. Decoder The Decoder is responsible for generating the vector used in the next word prediction component based on the output of the encoder and all the tokens in the current sequence.   The first token to be processed is <SOS> which means Start of Sentence. Just like before  this token will be encoded through word embedding  positional embedding and self-attention.   After that  a skip connection is applied to sum the combined output of the word and position embedding to the output of the self-attention cell. The outcome of this process will go through another self-attention cell to be combined with the output of the Encoder  followed again by a skip connection. Just like in the Encoder  this final vector goes through a FNN to generate the vector that feeds the final component of the architecture  next word prediction.   5. Next Word Prediction The final output of the decoder goes through a simple neural network that will convert this vector into a new one with the same dimension as the vocabulary size. After that  a Softmax function is applied to determine which word should come next  as shown in Figure 13. The process stops when a <EOS> token (end of sentence) is predicted.   Final remarks about Self-Attention:   Query  Keys and Values are calculated in parallel  i.e.  Q  K and V are calculated at the same time for each word.We can have several Self-Attention Cells to train different weights and capture different relationships between words (the original had 8 cells that are combined through concatenation followed by a linear layer).There are multiple stacked layers (N=6) of self-attention cells + FFN which means the second layer input is the output of the first layer  the third layer input is the second layer output and so on and so forth.After each self-attention cell and FFN block  a normalization step is applied.LLaMA 3  Mixtral and Gemma: Whats new?This section will cover what are the main novel concepts  introduced by Meta  Mistral AI and Google with its respective models  LLaMA [3] [4] [5]  Mixtral [6] and Gemma [7].   Vocabulary Size  Context Length and ArchitectureAs we have seen previously in this article  the vocabulary size depends on the tokenizer adopted to feed the transformer. The tokenizer used in the original Transformer architecture had a vocabulary size of 32 000 tokens which is the same as in Mixtral. However  Gemma and LLaMA 3 have different tokenizers with 256 000 and 128 000 tokens  respectively.   Apart from the vocabulary size  the context length has been increasing over time since it has been demonstrated that a bigger context length  i.e. more tokens processed per sequence  yield to improved model performance. The original Transformer had a context length of 512 tokens  while Gemma and LLaMA 3 have 8192 and Mixtral 4096.   Also  on contrary to the original architecture that is an encoder-decoder Transformer for language translation  these models are decoder-only since their primary goal is to generate text.   Positional EmbeddingsLLaMA 3 and Gemma use Rotary Positional Embedding (RoPE) instead of the original version. This approach brings benefits such as modelling the relative position between tokens which means that the tokens in position 1 and 2 will be more similar than the tokens in position 1 and 500.   For example  lets consider the sentence Gemma is better than LLaMA and a 2D word embedding. The positional embedding of the word better will be given by a rotation of the original vector based on position 3 and a constant .   If two new words are added to the beginning of the sentence  then the angle between better and than will keep the same  as shown in Figure 15.   Grouped Query Attention instead of Multi Head AttentionLLaMA 3  Gemma and Mixtral replaced the traditional Multi Head Attention with Grouped Query Attention for faster decoding  hence  faster inference.   GQA-G divides query values into G groups that share a single key and value head (GQA-1 = MQA while a GQA-H = MHA). This approach reduces the number of keys and values heads into a single key and value per query group  accelerating the inference speed and reducing the memory requirements during decoding with a quality closer to MHA.   Activation functionLLaMA 3 and Gemma replaced the traditional ReLU activation function in the FFN block with SwiGLU and GeGLU  respectively. These functions  unlike ReLU that converts all negative values to 0  have a parameter that smooths this conversion where the probability of setting negative values to 0 increases as these values are closer to 0.   SMoE: Sparse Mixture of ExpertsMixtral differs from the other architectures by using Mixture of Experts rather than stacking a FFN on top of the different attention cells. Each Expert is responsible for processing a type of token  for example  one can be a punctuation expert  a visual description expert  or a number expert.   The Expert(s) that is going to process the token is chosen by a Gated Network trained to perform this allocation. This approach bring benefits such as efficiency by activating less model parameters and more accurate predictions because each expert is focused on a specific task.   DSPy: an easy to use library to get started with LLMsDSPy is an interesting library built around Signatures to test LLMs in different contexts or problems. The Signature allows to ask a LLM to perform different tasks without much prompt engineering or changes in the code  for example  we can perform sentiment analysis by using the signature sentence -> sentiment or summarization by using document -> summary or a RAG system using context  question -> answer.   Besides that  you can create personalized Signatures with just a few lines of code and  also  retrieve the reasoning that led the LLM to provide a certain answer by using ChainOfThought class. But  lets see it on practice.   First step is to import the libraries and setting up an env file with the HuggingFace (HF) token and  if you have a Open AI key  you can also use it with DSPy.   %load_ext autoreload %autoreload 2 import os import dspy from dotenv import load_dotenv load_dotenv('env/var.env')After that  we can load any model in HF repository or you can call ChatGPT by running the following lines of code.   In our case  we used mistralai/Mistral-7B-Instruct-v0.1 from HF.   chatgpt=dspy.OpenAI(api_key=os.getenv(OPENAI_KEY)  model='gpt-3.5-turbo-1106') lm = dspy.HFModel(model='mistralai/Mistral-7B-Instruct-v0.1'  token=os.getenv(HF_TOKEN)) dspy.configure(lm=lm)Sentiment Analysis   The Signature that allows to perform sentiment analysis is sentence -> sentiment. In our example  we created three different sentences about how AI is helpful  how uncertain it is if it brings more advantages than disadvantages and  finally  how threatening it is in order to try to capture three different sentiments.   As you can see below  with three lines of code we managed to extract sentiment from sentences using Mistral.   # positive sentiment sentence = AI has the potential to enhance human capabilities and automate tedious tasks  thereby improving our productivity and quality of life. classify = dspy.Predict('sentence -> sentiment') result = classify(sentence=sentence) print(result.sentiment) # neutral sentiment sentence = Poorly designed or uncontrolled AI systems pose risks such as job displacement  privacy violations  and even existential threats if advanced AI becomes misaligned with human values. result = classify(sentence=sentence) print(result.sentiment) # negative sentiment sentence = AI can bring existential threats if it becomes misaligned with human values. result = classify(sentence=sentence) print(result.sentiment)Positive Neutral Negative   Personalized Signature   In this case  we will create a personalized signature that aims to make Mistral classify product return reasons based on a list of reasons and the customer explanation for the return.   For that  we create a class called ReasonList that inherits a DSPy Signature and  in the output field  we define which classes the LLM must use to classify the input sentence. The huge benefit of this approach is that with just one line of code  we can make the LLM provide a formatted answer.    # an example below of the a custom signature and with a defined output class ReasonList(dspy.Signature): Classify reason among no_need  too_large  too_small  does_not_look_good sentence = dspy.InputField() reason = dspy.OutputField(desc=A list of values of any one of no_need  too_large  too_small or does_not_look_good  format=list[str]) sentence = I'm returning this item because my sister offered me a similar one classify = dspy.Predict(ReasonList) result = classify(sentence=sentence) print(result.reason) sentence = I'm returning this item because it is not to my taste result = classify(sentence=sentence) print(result.reason)no_need does_not_look_good   Summarization   The Signature that allows to perform summarization is document -> summary. In our example  we provide a huge text about the advantages and disadvantages of AI  and we ask Mistral to summarize it for us without providing any prompt and letting DSPy to handle that for us.   # an example below of the same signature but with different modules document = AI technologies hold great promise to enhance and augment human capabilities in a wide range of domains. One of the key benefits of AI is its ability to automate routine and repetitive tasks  freeing up human workers to focus on more complex  creative  and strategic work. AI-powered automation can drive gains in productivity  efficiency  and cost savings across industries. Additionally  AI systems excel at quickly processing and analyzing large datasets  identifying patterns and insights that may elude human cognition. This data-driven decision making can lead to more informed  evidence-based choices in fields like healthcare  finance  transportation  and scientific research. For example  AI algorithms can assist doctors in early disease detection  optimize logistics and supply chains  and accelerate drug discovery. Furthermore  AI has transformative potential in enhancing human experiences. AI-powered personal assistants  chatbots  and recommender systems can provide personalized assistance  information  and content tailored to individual needs and preferences. This can improve customer service  education  and quality of life. Advancements in natural language processing  computer vision  and robotic technologies also hold promise for assisting the elderly and disabled  improving accessibility  and extending human physical capabilities. Lastly  AI may play a vital role in addressing global challenges  such as climate change  resource scarcity  and disease outbreaks. AI systems can help optimize energy grids  model climate patterns  accelerate scientific research  and coordinate disaster response efforts more efficiently than human-only approaches. Of course  the development and deployment of AI also come with important ethical considerations and potential risks that must be carefully navigated. But overall  the transformative potential of AI to augment and empower humanity is profound and worth continued responsible exploration and investment. summarize = dspy.Predict('document -> summary') response = summarize(document=document) print(response)AI technologies hold great promise to automate routine tasks  drive productivity and efficiency gains  and enable data-driven decision making across industries. They also have the potential to enhance human experiences  address global challenges  and extend human physical capabilities. However  the development and deployment of AI also come with important ethical considerations and potential risks that must be carefully navigated.   RAG Systems   The Signature that allows to implement a RAG system is context  question -> answer. In our example  we provide a context about how the stock market and company valuation has been fluctuating in the last few weeks and we ask the LLM to retrieve which is the most valuable company in the end of the fluctuation period which it did correctly.    # an example below of the a custom signature using the basic RAG structure. context =  Context: Nvidia's stock has nearly tripled so far this year  compared with a rise of about 19% in Microsoft shares  with demand for its top-of-the-line processors outpacing supply. Tech giants Microsoft  Meta Platforms (META.O)  opens new tab and Google-owner Alphabet (GOOGL.O)  opens new tab are competing to build out their AI computing capabilities and add the technology to their products and services. An insatiable appetite for Nvidia's AI processors  viewed as far superior to competitors' offerings  has left them in tight supply  and many investors view Nvidia as the greatest winner to date from surging AI development. Although in the last few weeks we have been seing NVIDIA reaching the top of the most valued companies in the world  it was surpassed by Apple after the news about their heavy investment in AI.  question = What is the most valuable company? qa = dspy.ChainOfThought('context  question -> answer'  temperature=0.7) response = qa(context=context  question=question) print(response.answer)As of the recent developments  Apple has surpassed Nvidia and is currently the most valuable company  following their heavy investment in AI.   Since  we used ChainOfThought  we can also retrieve what was the reasoning for the LLM to get to that answer by running the following code:   print(lm.inspect_history(n=1))Reasoning: Lets think step by step in order to produce the answer. We need to consider the recent news and developments in AI technology and its impact on the value of companies.   Answer: As of the recent developments  Apple has surpassed Nvidia and is currently the most valuable company  following their heavy investment in AI.   ConclusionLarge Language Models have been around since the release of ChatGPT in 2022 and it is important to understand how these models work in order to extract their full potential to improve the way we live.   In this article  we went from the basics to the advanced concepts of the Transformer architecture to deeply understand how it works and how it is able to generate text in such accurate manner. Apart from that  we also explored how LLMs have been evolving since 2017  with big companies like Meta and Google releasing their own models leveraging the Transformer architecture with novel concepts.   Finally  the practical implementation of these models have been facilitated by packages like DSPy that remove the overhead of developing specific solutions for each LLM and allow to quickly perform several experiments to determine which LLM is more suitable for our use case.   Keep in touch: LinkedIn  Medium   References[1] Kudo  Taku  and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226 (2018).   [2] Vaswani et al.  Attention Is All You Need  arXiv preprint arXiv:1706.03762 (2017).   [3] Touvron et al.  LLaMA: Open and Efficient Foundation Language Models  arXiv preprint arXiv:2302.13971 (2023).   [4] Touvron et al.  Llama 2: Open Foundation and Fine-Tuned Chat Models  arXiv preprint arXiv:2307.09288 (2023).   [5] https://ai.meta.com/blog/meta-llama-3/   [6] Albert et al.  Mixtral of Experts  arXiv preprint arXiv:2401.04088 (2024).   [7] Gemma Team  Google DeepMind  Gemma: Open Models Based on Gemini Research and Technology  arXiv preprint arXiv:2403.08295 (2023).