## Inspiration

We were inspired by Pokemon Go and Duolingo. We believe that exploring the world and learning a new language pair up well together. Associating new vocabulary with real experiences helps you memorize it better.

## What it does

* Explore your surroundings to find landmarks around you.
* To conquer a landmark you will have to find and photograph objects that represent the vocabulary words we assigned to the land mark.
* As you conquer landmarks you will level up and more advanced landmarks (with more advanced vocab) will be unlocked.
* Can you conquer all the landmarks and become a Vocab Master?

## How we built it

* We used react native for the app, we used react-native-maps for the map, we used expo camera for the camera, and we used Python for the backend.
* we used Google Cloud Vision API for object recognition and annotated images with identified key object labels
-we used Google Cloud Translate API to translate the names of identified objects to the user's selected target language.
* we used the Gemini API to generate useful questions based on identified objects in the picture the user takes.

## Challenges we ran into

-Stitching together our front and back end
-We ran into issues with Firebase deployment in particular, as with other Flask app hosting services

## Accomplishments that we're proud of

-Created a unique UI that interfaces with the camera on the user's mobile device
-Creating a landmark exploration map for users to embark on vocabulary challenges
-Creating a quiz functionality using react native that ensures users review their learned vocabulary regularly. This works by requiring users to select the correct translations of words and phrases from previous photos every fifth photo taken.
-Developing a Python backend that takes the URI of an image as input and returns an image annotated with key objects, as well as translations for those objects in selected target language and example sentences using the identified objects in the target language based on the user's surroundings.

## What we learned

-Repository branch organization is important (keep git good)
-Dividing tasks among teammates leads to more rapid progress, but integration can be challenging
-Prioritization is key, when one thing is not working it is better to get some working functionality given the time constraints of the Hackathon.

## What's next for Pocket Vocab by Purple Cow

-Polishing the UI and integration between all the services and ensuring features work seamlessly together
-Add additional features to increase social aspects of the app. Such as conquering landmarks by holding the record for largest vocabulary at a certain landmark
## DnD PvP Mode - AI-Generated Text-Based Adventure/ PvP Game

Inspiration
The inspiration for DnD PvP Mode stemmed from our love for traditional tabletop role-playing games like Dungeons & Dragons. We wanted to create an immersive and dynamic experience where players could engage in turn-based combat using natural language input, leveraging the power of AI to generate combat art through Dall-E 3.

## What it does

DnD PvP Mode is a text-based adventure and player versus player (PvP) game that brings the magic of Dungeons & Dragons into the digital realm. Two players enter the game from different computers, each choosing their character's name and class. The AI Dungeon Master (DM) takes charge, setting the scene with detailed descriptions of the environment and characters. Players are then free to interact with the world, unleashing their creativity and convincing the DM with their decisions.

The game's core revolves around turn-based combat, where players input their actions and strategies using natural language. The AI DM parses this input, orchestrating a thrilling battle that unfolds in the minds of the players. To enhance the immersive experience, we incorporated Dall-E 3, an AI model that generates stunning combat art based on the DM's descriptions, bringing the characters and scenes to life.

## How we built it

We used Python as our primary programming language, making heavy use of the OpenAI API for GPT3.5 to power the AI Dungeon Master. This intelligence enabled it to understand and respond to the players' unrestrained natural language input and parse out function calls to progress the game. To generate captivating combat art, we integrated Dall-E 3 into the game flow, turning textual descriptions into striking images.

We used Flask for the web server (along with HTML, CSS, and JavaScript) to create an intuitive and responsive user experience. Flask's simplicity let us rapidly prototype our game to create the best version of it we could as fast as we could. The web-based design also let players join in the game from their own devices, allowing them to make their decisions and witnessing the unfolding narrative from anywhere they chose.

## Challenges we ran into

One of the main challenges we faced was parsing and integrating user input into a solid underlying game system. Using GPT functions to parse natural language output and consistently produce quality outputs was a challenging task, but rewarding. Coordinating the communication between the AI Dungeon Master, Dall-E 3, and the web interface took a lot of work, especially to guarantee that each player's client remained in-sync and avoided race conditions.
Ensuring that the generated combat art aligned with the narrative and remained stylistically consistent between prompts posed a creative and technical challenge that required a lot of careful prompt engineering. Although we managed to stabilize the style, we weren't able to generate consistent characters between turns (although this could likely be done with fine-tuned models trained to output only a few characters). Additionally, although AI generated art has improved drastically as of late, it still lacks a human touch and direction to it, and each image does not necessarily play well into the next as it would in a hand-drawn and planned story. Further, dungeon masters usually make funny voices, and ours doesn't.

## Accomplishments that we're proud of

We are proud to have successfully combined traditional tabletop RPG elements with cutting-edge AI technology. The ability to have our unrestricted natural-language input consistently parsed into game terms and rapidly illustrated is a joy and a significant achievement for our team.

## What we learned

Throughout the hackathon, we learned a great deal about using AI for interactive storytelling and visual art generation, as well as a first foray into game design. Also, the effective integration of GPT3.5 and Dall-E 3 required us to attain a deeper level of understanding of both technologies. Having made this prototype with these tools, we are excited to continue exploring their potential in future projects.

## What's next for DnD PvP Mode

The journey doesn't end here! In the future, we plan to expand DnD PvP Mode by incorporating more status features, classes, and environments. We aim to enhance the AI Dungeon Master's capabilities with a broader range of functions to call, allowing for even more intricate and personalized storytelling and statuses that it may inflict upon the players. In future, we may also be able to create stable characters which the AI art generation models may be able to recreate in each scene for a consistent visual story to pair with the text. Stay tuned for more adventures in the world of DnD PvP Mode!
## Inspiration

Imagine a world where your best friend is standing in front of you, but you can't see them. Or you go to read a menu, but you are not able to because the restaurant does not have specialized brail menus. For millions of visually impaired people around the world, those are not hypotheticals, they are facts of life.

Hollywood has largely solved this problem in entertainment. Audio descriptions allow the blind or visually impaired to follow the plot of movies easily. With Sight, we are trying to bring the power of audio description to everyday life.

## What it does

Sight is an app that allows the visually impaired to recognize their friends, get an idea of their surroundings, and have written text read aloud. The app also uses voice recognition to listen for speech commands to identify objects, people or to read text.

## How we built it

The front-end is a native iOS app written in Swift and Objective-C with XCode. We use Apple's native vision and speech API's to give the user intuitive control over the app.

---

The back-end service is written in Go and is served with NGrok.

---

We repurposed the Facebook tagging algorithm to recognize a user's friends. When the Sight app sees a face, it is automatically uploaded to the back-end service. The back-end then "posts" the picture to the user's Facebook privately. If any faces show up in the photo, Facebook's tagging algorithm suggests possibilities for who out of the user's friend group they might be. We scrape this data from Facebook to match names with faces in the original picture. If and when Sight recognizes a person as one of the user's friends, that friend's name is read aloud.

---

We make use of the Google Vision API in three ways:

* To run sentiment analysis on people's faces, to get an idea of whether they are happy, sad, surprised etc.
* To run Optical Character Recognition on text in the real world which is then read aloud to the user.
* For label detection, to indentify objects and surroundings in the real world which the user can then query about.

## Challenges we ran into

There were a plethora of challenges we experienced over the course of the hackathon.

1. Each member of the team wrote their portion of the back-end service a language they were comfortable in. However when we came together, we decided that combining services written in different languages would be overly complicated, so we decided to rewrite the entire back-end in Go.
2. When we rewrote portions of the back-end in Go, this gave us a massive performance boost. However, this turned out to be both a curse and a blessing. Because of the limitation of how quickly we are able to upload images to Facebook, we had to add a workaround to ensure that we do not check for tag suggestions before the photo has been uploaded.
3. When the Optical Character Recognition service was prototyped in Python on Google App Engine, it became mysteriously rate-limited by the Google Vision API. Re-generating API keys proved to no avail, and ultimately we overcame this by rewriting the service in Go.

## Accomplishments that we're proud of

Each member of the team came to this hackathon with a very disjoint set of skills and ideas, so we are really glad about how well we were able to build an elegant and put together app.

Facebook does not have an official algorithm for letting apps use their facial recognition service, so we are proud of the workaround we figured out that allowed us to use Facebook's powerful facial recognition software.

We are also proud of how fast the Go back-end runs, but more than anything, we are proud of building a really awesome app.

## What we learned

Najm taught himself Go over the course of the weekend, which he had no experience with before coming to YHack.

Nathaniel and Liang learned about the Google Vision API, and how to use it for OCR, facial detection, and facial emotion analysis.

Zak learned about building a native iOS app that communicates with a data-rich APIs.

We also learned about making clever use of Facebook's API to make use of their powerful facial recognition service.

Over the course of the weekend, we encountered more problems and bugs than we'd probably like to admit. Most of all we learned a ton of valuable problem-solving skills while we worked together to overcome these challenges.

## What's next for Sight

If Facebook ever decides to add an API that allows facial recognition, we think that would allow for even more powerful friend recognition functionality in our app.

Ultimately, we plan to host the back-end on Google App Engine.