fixed requirements

app.py

@@ -11,6 +11,9 @@ file2 = st.file_uploader("Upload a file", type=["md", "txt"], key="second")
 topics = {}
 results = {}
 
+embedder = utils.load_model()
+nlp = utils.load_nlp()
+
 if file1 is not None and file2 is not None:
     
     input_text1 = file1.read().decode("utf-8")
@@ -32,9 +35,7 @@ if file1 is not None and file2 is not None:
         topics['insight2'] = [keywords2, concepts2]
         
     with st.spinner("Flux capacitor is fluxing..."):
-        embedder = utils.load_model()
-        clutered = utils.cluster_based_on_topics(embedder, cleaned_text1, cleaned_text2, num_clusters=5)
-        # print(clutered)
+        clutered = utils.cluster_based_on_topics(nlp, embedder, cleaned_text1, cleaned_text2, num_clusters=3)
         
     with st.spinner("Polishing up"):
         results = utils.generate_insights(topics, file1.name, file2.name, cleaned_text1, cleaned_text2, clutered)

prompter/insights_33.prompt

@@ -1,41 +0,0 @@
-You are a highly intelligent bot that is tasked with common ideas between documents. The following are two documents that have been topic modelled and have been clustered based on concepts.
-
-The name for document 1 is : Good conversations have lots of doorknobs.md
-
-The name for document 2 is : First we shape our social graph; then it shapes us.md
-
-The topics for document 1 is : spiderman,singing,musical,chorus,song,singing something,about spiderman,spiderman spiderman,spiderman,spiderman and
-
-The topics for document 2 is : chimpanzees,genetically,womb,consciously,upbringings,from chimpanzees,chimpanzees as,chimpanzees in,chimpanzees and,chimpanzees
-
-The more complex concepts in document 1 is : singing like spiderman,spiderman sudden pianist,songs spiderman scientific,just songs spiderman,excerpt spiderman boyfriend
-
-The more complex concepts in document 2 is : chimpanzees born habitat,die chimpanzees born,sets apart chimpanzees,fast die chimpanzees,old children chimpanzees
-
-The sentences in one of the clusters is : ask remove mask spot really hard, trick kept us afloat called “take-and-take focus,” meaning whoever singing keep going someone jumped take spotlight them, happen quickly often.
-it’s easy remember lonely feels taker refuses cede spotlight you, easy forget lovely feels don’t want spotlight taker lets recline mezzanine fill stage.
-it’s often unclear, stand around waiting someone else take turn invite us take ours.
-we’re standing perimeter empty dance circle, takers martyrs launch middle .
-
-From the sentences and topics above, explain the common idea between the documents and write a paragraph about it and give me 3 new concepts that are linked to this idea. 
-You output format should be:
-
-"""
-name: <FILL-CONCEPT-NAME-HERE>
-description: <FILL-CONCEPT-DESCRIPTION-HERE>
-related:
-- <FILL-RELATED-CONCEPT-1>
-- <FILL-RELATED-CONCEPT-2>
-- <FILL-RELATED-CONCEPT-3>
-"""
-
-The common idea between the documents is the importance of collaboration and teamwork. In the first document, the idea of collaboration is explored in the context of music, with the chorus singing together to create a beautiful song. In the second document, the idea of collaboration is explored in the context of chimpanzees, with the idea that they work together to survive and thrive in their environment. 
-
-The concept of collaboration is an important one, and it is essential for any group of individuals to work together to achieve a common goal.
-
-name: Group Dynamics
-description: Group dynamics is the study of how people interact in groups and how their behavior affects the group as a whole.
-related:
-- Interpersonal Relationships
-- Social Interaction
-- Conflict Resolution

requirements.txt

@@ -1,8 +1,10 @@
 keybert==0.7.0
-mdforest==1.5.0
+mdforest==1.5.4
 nltk==3.8.1
+numpy==1.23.5
 openai==0.27.2
-pandas==1.5.3
+scikit_learn==1.2.2
+en_core_web_sm
 sentence_transformers==2.2.2
 spacy==3.5.2
 streamlit==1.21.0

utils.py

@@ -5,6 +5,7 @@ from transformers import AutoTokenizer
 import os, re, json
 import openai
 import spacy
+import en_core_web_sm
 from sklearn.cluster import KMeans, AgglomerativeClustering
 import numpy as np
 from sentence_transformers import SentenceTransformer
@@ -27,6 +28,11 @@ def load_model():
 	embedder = SentenceTransformer(MODEL)
 	return embedder 
 
+@st.cache_data
+def load_nlp():
+	nlp = en_core_web_sm.load()
+	return nlp
+
 def create_nest_sentences(document:str, token_max_length = 1023):
 	nested = []
 	sent = []
@@ -66,9 +72,9 @@ def generate_keywords(kw_model, document: str) -> list:
 				final_topics.append(extraction[0])
 		return final_topics
 
-def cluster_based_on_topics(embedder, text1:str, text2:str, num_clusters=3):
-	nlp = spacy.load("en_core_web_sm")
-	
+
+def cluster_based_on_topics(nlp, embedder, text1:str, text2:str, num_clusters=3):
+		
 	# Preprocess and tokenize the texts
 	doc1 = nlp(preprocess(text1))
 	doc2 = nlp(preprocess(text2))