## Inspiration

The whiteboard or chalkboard is an essential tool in instructional settings - to learn better, students need a way to directly transport code from a non-text medium to a more workable environment.

## What it does

Enables someone to take a picture of handwritten or printed text converts it directly to code or text on your favorite text editor on your computer.

## How we built it

On the front end, we built an app using Ionic/Cordova so the user could take a picture of their code. Behind the scenes, using JavaScript, our software harnesses the power of the Google Cloud Vision API to perform intelligent character recognition (ICR) of handwritten words. Following that, we applied our own formatting algorithms to prettify the code. Finally, our server sends the formatted code to the desired computer, which opens it with the appropriate file extension in your favorite IDE. In addition, the client handles all scripting of minimization and fileOS.

## Challenges we ran into

The vision API is trained on text with correct grammar and punctuation. This makes recognition of code quite difficult, especially indentation and camel case. We were able to overcome this issue with some clever algorithms. Also, despite a general lack of JavaScript knowledge, we were able to make good use of documentation to solve our issues.

## Accomplishments that we're proud of

A beautiful spacing algorithm that recursively categorizes lines into indentation levels.
Getting the app to talk to the main server to talk to the target computer.
Scripting the client to display final result in a matter of seconds.

## What we learned

How to integrate and use the Google Cloud Vision API.
How to build and communicate across servers in JavaScript.
How to interact with native functions of a phone.

## What's next for Codify

It's feasibly to increase accuracy by using the Levenshtein distance between words. In addition, we can improve algorithms to work well with code. Finally, we can add image preprocessing (heighten image contrast, rotate accordingly) to make it more readable to the vision API.
## Inspiration

Imagine a world where your best friend is standing in front of you, but you can't see them. Or you go to read a menu, but you are not able to because the restaurant does not have specialized brail menus. For millions of visually impaired people around the world, those are not hypotheticals, they are facts of life.

Hollywood has largely solved this problem in entertainment. Audio descriptions allow the blind or visually impaired to follow the plot of movies easily. With Sight, we are trying to bring the power of audio description to everyday life.

## What it does

Sight is an app that allows the visually impaired to recognize their friends, get an idea of their surroundings, and have written text read aloud. The app also uses voice recognition to listen for speech commands to identify objects, people or to read text.

## How we built it

The front-end is a native iOS app written in Swift and Objective-C with XCode. We use Apple's native vision and speech API's to give the user intuitive control over the app.

---

The back-end service is written in Go and is served with NGrok.

---

We repurposed the Facebook tagging algorithm to recognize a user's friends. When the Sight app sees a face, it is automatically uploaded to the back-end service. The back-end then "posts" the picture to the user's Facebook privately. If any faces show up in the photo, Facebook's tagging algorithm suggests possibilities for who out of the user's friend group they might be. We scrape this data from Facebook to match names with faces in the original picture. If and when Sight recognizes a person as one of the user's friends, that friend's name is read aloud.

---

We make use of the Google Vision API in three ways:

* To run sentiment analysis on people's faces, to get an idea of whether they are happy, sad, surprised etc.
* To run Optical Character Recognition on text in the real world which is then read aloud to the user.
* For label detection, to indentify objects and surroundings in the real world which the user can then query about.

## Challenges we ran into

There were a plethora of challenges we experienced over the course of the hackathon.

1. Each member of the team wrote their portion of the back-end service a language they were comfortable in. However when we came together, we decided that combining services written in different languages would be overly complicated, so we decided to rewrite the entire back-end in Go.
2. When we rewrote portions of the back-end in Go, this gave us a massive performance boost. However, this turned out to be both a curse and a blessing. Because of the limitation of how quickly we are able to upload images to Facebook, we had to add a workaround to ensure that we do not check for tag suggestions before the photo has been uploaded.
3. When the Optical Character Recognition service was prototyped in Python on Google App Engine, it became mysteriously rate-limited by the Google Vision API. Re-generating API keys proved to no avail, and ultimately we overcame this by rewriting the service in Go.

## Accomplishments that we're proud of

Each member of the team came to this hackathon with a very disjoint set of skills and ideas, so we are really glad about how well we were able to build an elegant and put together app.

Facebook does not have an official algorithm for letting apps use their facial recognition service, so we are proud of the workaround we figured out that allowed us to use Facebook's powerful facial recognition software.

We are also proud of how fast the Go back-end runs, but more than anything, we are proud of building a really awesome app.

## What we learned

Najm taught himself Go over the course of the weekend, which he had no experience with before coming to YHack.

Nathaniel and Liang learned about the Google Vision API, and how to use it for OCR, facial detection, and facial emotion analysis.

Zak learned about building a native iOS app that communicates with a data-rich APIs.

We also learned about making clever use of Facebook's API to make use of their powerful facial recognition service.

Over the course of the weekend, we encountered more problems and bugs than we'd probably like to admit. Most of all we learned a ton of valuable problem-solving skills while we worked together to overcome these challenges.

## What's next for Sight

If Facebook ever decides to add an API that allows facial recognition, we think that would allow for even more powerful friend recognition functionality in our app.

Ultimately, we plan to host the back-end on Google App Engine.
*"There are these two young fish swimming along and they happen to meet an older fish swimming the other way, who nods at them and says “Morning, boys. How’s the water?” And the two young fish swim on for a bit, and then eventually one of them looks over at the other and goes “What the hell is water?”"*
-David Foster Wallace, *This is Water*

Often, the most important things about our lives are the ones that become hidden in plain sight. Like water to a fish, our perceptions of ourselves, our loved ones, and the world around us are often steeped in intentions we have trouble recognizing.

## We want to change how journaling works.

We want to help users reflect by applying LLM AI technology to their journaling. Afterthought picks up on your intention (whether you're speaking about a goal, an experience, a relationship, or something coming up) as well as your mood in order to give you a benchmark to better understand yourself. Pairing this analysis with a summary of recent journals and reflection prompts, we intend to make journaling go deeper. Not only does Afterthought inform you of patterns, it actively prompts you to question the things you've been telling yourself.

## How we built it

We started out in the cohere playground, brainstorming what kind of categories would be valuable. We went into our own notes apps and journals to see how we might use the app. From there, we began constructing a web app, a categorize script, and a summarize script. We later married these elements together.

Our dataset was first formed by coming up with out own examples of a variety of possible notes. Then, we asked chatGPT to come up with a variety of similar notes, varying the way these sentences were structured and the names mentioned. We pruned this list to make sure it was a balanced representation of possible notes.

## Challenges we experienced

Nobody on our team had used all of these technologies before, meaning that there was something we all learned. In addition to the typical hackathon blues (drowsiness and sore tummies), we spent a lot of time getting around learning curves.

## What's next

We'd like to add features that allow for people to track specific relationships/concepts, rather than these general categories. Additionally, creating a way to classify and reflect upon ideas/art is a direction we'd be interested in exploring.