## What it does

YapTrack is a real-time meeting guide that forces you to close the loop on all of your ideas

## Inspiration

One of our team members spent 5 hours in a meeting just to MOVE SOME COLUMNS IN A DATABASE. The meeting should have been two hours, but it got stretched out because every time we proposed a small changes, we weren’t able to keep track of all of the prior requirements to our solution and the resulting implications of the change. This resulted in a feedback loop where we kept following the chain of ideas, forgetting why we changed something and reverting back and forth. 5 Software engineers, getting paid about $500 in company resources per an hour, got dragged out for 3 extra hours ($1500 in unnecessary costs) just to make a decision that they had considered from the start.

Topic fluctuations constantly arise in our meetings. We are tired of wasting our time trying to remember how to circle back to a concrete starting point. So we intended to solve this problem by creating a meeting guide that highlights questions that have yet to be explored to completion and previous decisions that informed our decision making process.

## How we built it

We take speech audio, and in real-time convert it into text. In order to provide a real time speech experience with Groq (which requires complete audio files) we needed to send many small audio segments to create real-time transcriptions of spoken language. As a result, we constantly sliced our audio stream on 5 second increments. We queried LLMS to predict the intended sentence structure when the splits occurred in the middle of a word. For example, when the splitted language looked like “A”, “wire”, the LLM could predict that we intended to say “Acquire”. Thus using assistive LLM merging on different segments seamlessly built upon our ideas as we gained more speech-to-text information. As we keep adding information, we also use llama-index LLM to extract entity-relationship-entity triplets for building nodes and edges in our knowledge graph. Once our knowledge graph is built, we visualize current idea branches and the full knowledge graph. Additionally, the entities can be re-evaluated for additional correlation edges in our graph, by using a transformer model to embed the entities and threshold cosine similarity scores.

For example: when the splitted language looked like “A”, “wire”. Using the context of the sentence, the LLM could predict that we intended to say “Acquire”. Thus using assistive LLM merging on different segments seamlessly built upon our ideas as we gained more speech-to-text information.

For the backend, a graphical database, Neo4j, was used to keep track of both our initial node relationships based on the text, and also additional correlations derived after generating the knowledge graph.

Repo - Next JS w/ Flask and Aceternity UI ← need more info here

## Challenges we ran into

Similarity Search and Similarity Thresholding: When generating different entities and relationships using our LLM, we realized that some nodes might be very closely correlated, but not understood in a sentence structure. Therefore, we used the BERT (Bidirectional Encoder Representations from Transformers) model, a pre-trained language model developed by Google. It gives us semantic encoding of all our entities, which allows us to do cosine similarities on our embeddings, and find new correlations on our knowledge graph. Based on a threshold, we create correlation edges on the graph. This could be potentially useful for finding new relationships or merging highly related nodes that represent near similar ideas.
Redundancies in Knowledge graph entity generation . We queried LLAMA for (Source Entity -> Edge -> Destination Entity) mappings. Though it immediately began to form pairings, it often created high similarity pairings like (“greg”, “likes”, “sleeping on the couch”) and (“greg”, “enjoys”, “spending time sleeping on the couch”) which is a redundancy that we wanted to avoid when querying the graph. We used three techniques to mitigate these redundancies:
GraphDB input to our RAG queries provided context for matching new entities to closely related existing ones.
Prompt engineering using examples of closely matching inputs being generalized into the same entities
Vector Cosine Similarity Scores: Depending on the use-case of our knowledge graph (ie. Learning, Brainstorming, Requirements Analysis) we set different thresholds based
Knowledge graph real time visual display
Because the knowledge graph was dynamically being changed due to updates from real time audio streaming, it was difficult keeping the UI live. To address this issue, we had to extensively research React components and their operations.

## Accomplishments that we're proud of

Making use of the Neo4j Graph Database, allowing us to query data using the Cypher query language to extract paths and edges with less logic on the backend, and more logic in-database.
Using Groq and llama-index to quickly streamline speech recognition to a knowledge graph database.
Leveraging prompt engineering to get full power from our LLMs.
Finding more discrete relationships using transformer model embedding on text entities.

## What we learned

Learning how to use Groq (speech recognition and llama index node extractions).
Fine-tuning LLM prompts for specific output formats.
Embedding text information using machine learning.

## What's next for YapTrack

Adding a large list of prompt options for knowledge graph styles, or allowing users to make their own prompts for knowledge graph entities to fit their specific needs.
Merging nodes based on correlation calculations on similarity search.
## Inspiration

As software engineers, we constantly seek ways to optimize efficiency and productivity. While we thrive on tackling challenging problems, sometimes we need assistance or a nudge to remember that support is available. Our app assists engineers by monitoring their states and employs Machine Learning to predict their efficiency in resolving issues.

## What it does

Our app leverages LLMs to predict the complexity of GitHub issues based on their title, description, and the stress level of the assigned software engineer. To gauge the stress level, we utilize a machine learning model that examines the developer’s sleep patterns, sourced from TerraAPI. The app provides task completion time estimates and periodically checks in with the developer, suggesting when to seek help. All this is integrated into a visually appealing and responsive front-end that fits effortlessly into a developer's routine.

## How we built it

A range of technologies power our app. The front-end is crafted with Electron and ReactJS, offering compatibility across numerous operating systems. On the backend, we harness the potential of webhooks, Terra API, ChatGPT API, Scikit-learn, Flask, NodeJS, and ExpressJS. The core programming languages deployed include JavaScript, Python, HTML, and CSS.

## Challenges we ran into

Constructing the app was a blend of excitement and hurdles due to the multifaceted issues at hand. Setting up multiple webhooks was essential for real-time model updates, as they depend on current data such as fresh Github issues and health metrics from wearables. Additionally, we ventured into sourcing datasets and crafting machine learning models for predicting an engineer's stress levels and employed natural language processing for issue resolution time estimates.

## Accomplishments that we're proud of

In our journey, we scripted close to 15,000 lines of code and overcame numerous challenges. Our preliminary vision had the front end majorly scripted in JavaScript, HTML, and CSS — a considerable endeavor in contemporary development. The pinnacle of our pride is the realization of our app, all achieved within a 3-day hackathon.

## What we learned

Our team was unfamiliar to one another before the hackathon. Yet, our decision to trust each other paid off as everyone contributed valiantly. We honed our skills in task delegation among the four engineers and encountered and overcame issues previously uncharted for us, like running multiple webhooks and integrating a desktop application with an array of server-side technologies.

## What's next for TBox 16 Pro Max (titanium purple)

The future brims with potential for this project. Our aspirations include introducing real-time stress management using intricate time-series models. User customization options are also on the horizon to enrich our time predictions. And certainly, front-end personalizations, like dark mode and themes, are part of our roadmap.
## What it does

"ImpromPPTX" uses your computer microphone to listen while you talk. Based on what you're speaking about, it generates content to appear on your screen in a presentation in real time. It can retrieve images and graphs, as well as making relevant titles, and summarizing your words into bullet points.

## How We built it

Our project is comprised of many interconnected components, which we detail below:

#### Formatting Engine

To know how to adjust the slide content when a new bullet point or image needs to be added, we had to build a formatting engine. This engine uses flex-boxes to distribute space between text and images, and has custom Javascript to resize images based on aspect ratio and fit, and to switch between the multiple slide types (Title slide, Image only, Text only, Image and Text, Big Number) when required.

#### Voice-to-speech

We use Google’s Text To Speech API to process audio on the microphone of the laptop. Mobile Phones currently do not support the continuous audio implementation of the spec, so we process audio on the presenter’s laptop instead. The Text To Speech is captured whenever a user holds down their clicker button, and when they let go the aggregated text is sent to the server over websockets to be processed.

#### Topic Analysis

Fundamentally we needed a way to determine whether a given sentence included a request to an image or not. So we gathered a repository of sample sentences from BBC news articles for “no” examples, and manually curated a list of “yes” examples. We then used Facebook’s Deep Learning text classificiation library, FastText, to train a custom NN that could perform text classification.

#### Image Scraping

Once we have a sentence that the NN classifies as a request for an image, such as “and here you can see a picture of a golden retriever”, we use part of speech tagging and some tree theory rules to extract the subject, “golden retriever”, and scrape Bing for pictures of the golden animal. These image urls are then sent over websockets to be rendered on screen.

#### Graph Generation

Once the backend detects that the user specifically wants a graph which demonstrates their point, we employ matplotlib code to programmatically generate graphs that align with the user’s expectations. These graphs are then added to the presentation in real-time.

#### Sentence Segmentation

When we receive text back from the google text to speech api, it doesn’t naturally add periods when we pause in our speech. This can give more conventional NLP analysis (like part-of-speech analysis), some trouble because the text is grammatically incorrect. We use a sequence to sequence transformer architecture, *seq2seq*, and transfer learned a new head that was capable of classifying the borders between sentences. This was then able to add punctuation back into the text before the rest of the processing pipeline.

#### Text Title-ification

Using Part-of-speech analysis, we determine which parts of a sentence (or sentences) would best serve as a title to a new slide. We do this by searching through sentence dependency trees to find short sub-phrases (1-5 words optimally) which contain important words and verbs. If the user is signalling the clicker that it needs a new slide, this function is run on their text until a suitable sub-phrase is found. When it is, a new slide is created using that sub-phrase as a title.

#### Text Summarization

When the user is talking “normally,” and not signalling for a new slide, image, or graph, we attempt to summarize their speech into bullet points which can be displayed on screen. This summarization is performed using custom Part-of-speech analysis, which starts at verbs with many dependencies and works its way outward in the dependency tree, pruning branches of the sentence that are superfluous.

#### Mobile Clicker

Since it is really convenient to have a clicker device that you can use while moving around during your presentation, we decided to integrate it into your mobile device. After logging into the website on your phone, we send you to a clicker page that communicates with the server when you click the “New Slide” or “New Element” buttons. Pressing and holding these buttons activates the microphone on your laptop and begins to analyze the text on the server and sends the information back in real-time. This real-time communication is accomplished using WebSockets.

#### Internal Socket Communication

In addition to the websockets portion of our project, we had to use internal socket communications to do the actual text analysis. Unfortunately, the machine learning prediction could not be run within the web app itself, so we had to put it into its own process and thread and send the information over regular sockets so that the website would work. When the server receives a relevant websockets message, it creates a connection to our socket server running the machine learning model and sends information about what the user has been saying to the model. Once it receives the details back from the model, it broadcasts the new elements that need to be added to the slides and the front-end JavaScript adds the content to the slides.

## Challenges We ran into

* Text summarization is extremely difficult -- while there are many powerful algorithms for turning articles into paragraph summaries, there is essentially nothing on shortening sentences into bullet points. We ended up having to develop a custom pipeline for bullet-point generation based on Part-of-speech and dependency analysis.
* The Web Speech API is not supported across all browsers, and even though it is "supported" on Android, Android devices are incapable of continuous streaming. Because of this, we had to move the recording segment of our code from the phone to the laptop.

## Accomplishments that we're proud of

* Making a multi-faceted application, with a variety of machine learning and non-machine learning techniques.
* Working on an unsolved machine learning problem (sentence simplification)
* Connecting a mobile device to the laptop browser’s mic using WebSockets
* Real-time text analysis to determine new elements

## What's next for ImpromPPTX

* Predict what the user intends to say next
* Scraping Primary sources to automatically add citations and definitions.
* Improving text summarization with word reordering and synonym analysis.