The simple pitch behind ResQ: disaster recovery and evacuation are hard.

People crowd the streets making it difficult to quickly evacuate an area. Additionally, for those who choose not to evacuate, they face the possibility of being difficult to find for rescuers. What we've built is a 3 pronged approach.

1. ResQ Responder: An android application for rescuers that presents them with a triage list prepared by the ResQ ML Engine so as to attempt to save the most lives in the most efficient order. We also provide the ability to view this triage list in AR, making it easy to spot evacuees stuck on roofs or in hard to see places.
2. ResQ: The user-facing application. Simplicity is the goal. Only asking for user information to create a medical ID and rescue profile we use the application to record their GPS coordinates for rescue, as well as present them with push notifications about impending rescue. An evacuee can also use the application to broadcast his/her location to others.
3. ResQ ML Engine: The algorithms behind the ResQ platform. These allow us to effectively rank, triage and save victims while minimizing loss of life.
## Inspiration

I've always been fascinated by the complexities of UX design, and this project was an opportunity to explore an interesting mode of interaction. I drew inspiration from the futuristic UIs that movies have to offer, such as Minority Report's gesture-based OS or Iron Man's heads-up display, Jarvis.

## What it does

Each window in your desktop is rendered on a separate piece of paper, creating a tangible version of your everyday computer. It is fully featured desktop, with specific shortcuts for window management.

## How I built it

The hardware is combination of a projector and a webcam. The camera tracks the position of the sheets of paper, on which the projector renders the corresponding window. An OpenCV backend does the heavy lifting, calculating the appropriate translation and warping to apply.

## Challenges I ran into

The projector was initially difficulty to setup, since it has a fairly long focusing distance. Also, the engine that tracks the pieces of paper was incredibly unreliable under certain lighting conditions, which made it difficult to calibrate the device.

## Accomplishments that I'm proud of

I'm glad to have been able to produce a functional product, that could possibly be developed into a commercial one. Furthermore, I believe I've managed to put an innovative spin to one of the oldest concepts in the history of computers: the desktop.

## What I learned

I learned lots about computer vision, and especially on how to do on-the-fly image manipulation.
## Inspiration

On average, math education is—to put it lightly—incredibly poor. And to make matters worse, *good quality* math education is only accessible to a select few. When someone is faced with a homework question that they don't know how to answer, they usually just turn to the internet. While online calculators can get you a correct answer, they don't help you actually **learn** the material while you do so; showing their steps can only do so much.

Learning is a much more active process, something that is best done with the help of a good teacher or tutor. But, of course, quality tutoring is completely inaccessible to most. Our goal? We want to re-build how we do math help online, making high quality education free and accessible to all.

## What is CalculusCat?

CalculusCat is a two part process. The core is a math tutor, who takes the form of a cat: Sir Isaac Meowton. Meowton is built on a large language model, which we specifically engineered and fine-tuned to be accurate and helpful for guiding a student towards solutions. We do so by adopting an "active questioning" method, where learning is best done not by seeing the solution, but by engaging in conversation with the problem. Meowton is specifically trained to not provide the answer to a problem, but instead to determine gaps in your knowledge and help you fill them through active questioning.

In addition to Meowton, we wanted to take advantage of some of the great—but, sometimes overly general—resources available on the internet. We hand curated a couple quality sources, scraped the data off of them, and used transfomer-based text summarization and semantic search to send these resources to you while you're interacting with Meowton. This way, you can get a better grasp of the underlying concepts in the problem your solving, through resources specifically designed for quality teaching.

## How We Made It

We built the web app using Vercel, interfacing with the OpenAI API to access a GPT-3 model. We first fine-tuned this model on auto-generated arithmetic data, as well as auto-generated derivatives, in order to improve the models basic math capabilities. We then focused on **prompt engineering**, which involved writing a "scratchpad" as well as few-shot prompting the model before it was used. Scratchpads notably achieve higher accuracy than pure few-shot prompting, and our scratchpad allowed us to force CalculusCat to never directly provide the answer, instead only asking questions and providing guidance.

For the internet resource integration, we first chose a couple sources whom we know are useful for math tutoring (Khan Academy and 3Blue1Brown). Then, we scraped video transcripts from these two sources, compiling them into a large dataset. We used a pre-trained T5 model to summarize each of these transcripts, giving us shorter versions of each document; this prevented "extraneous" material (i.e. tangents) from confusing the semantic search. We then used a bidirectional transformer model to do semantic search based on sentence embeddings, taking CalculusCat's output as the query and using the entire body of transcripts as our corpus. Once we did this, we used a simple weighted average to score each summary based on how relevant each sentence was to the query, giving us a final ranking of videos. We then pass this video to the model for a recommendation.

## Learning

This project was a huge undertaking for us, since only a couple of us have much experience with programming, and none of us have ever built a web app end-to-end like this. None of us have used Vercel in the past, so we had to learn how to do that over the course of the hackathon, and only one of us has experience working with APIs. A lot of our difficulty was purely just technical integration, so we each worked on the areas we had the most experience with, which made it especially exciting to see it all come together in the end.

## Future Direction

There are a lot of places we want to take CalculusCat. One of the biggest is improving CalculusCat's math capabilities; currently, it's built on the davinci model of GPT-3, which is capable of doing math but is not infallible. In a way, CalculusCat sidesteps this issue by never directly attempting to solve a problem, which means all it needs to do is determine whether or not the user is correct. It does this with relatively high accuracy, but we'd still like to ensure that the backend model is really truly accurate in doing math.

Another big step is expanding CalculusCat's capability to beyond calculus, which we've currently restricted it to. This isn't too complex to execute, but it's definitely an important step in CalculusCat's journey to become a truly high-quality generalized tutor.

We'd also like to include more integrations with other math tools, like WolframAlpha, Desmos Graphing Calculator, and GeoGebra 3D. We'd also like to integrate a widget which would allow users to typeset their questions in LaTeX using MathQuill.