## Inspiration

People think of education simply as the encoding of knowledge, but most people don't realize that the hard part about learning is the recall. Our focus was to use Augmented Reality to enhance recall by mapping concepts and knowledge onto spatial objects on a path–a popular recalling technique called the Method of Loci. This idea, inspired from the Mind Palace technique in the tv show Sherlock, helps enhance association of knowledge using paths that we are familiar with - for instance, those we walk as we go through our daily routine.

## What it does

LociAR is an iOS App that helps users improve recall by enabling them to create virtual "mind palaces" based on routes they are familiar with. This is done through Augmented Reality - the user can create nodes corresponding to spatial objects on the path they choose to travel by, such as trees, lampposts, etc. Each node can store a title (visible through the camera), and a more detailed description that appears if the node is selected. The user can also store and load different paths, with each corresponding to any particular category.

## How we built it

We used ARKit along with the SceneKit framework to build the iOS App. Since none of us were familiar with Swift, we first learned the basics of iOS App Development and then created a flowchart for all the functions in the project. We each implemented various functions, such as focusing on the persistence of AR Sessions, creation of nodes, detection of nodes, etc.

## Challenges we ran into

There wasn't much detailed information about the technicalities of using ARKit with Xcode and as such, initial research was laborious. Moreover, AR technology is still very much in development and the tracking and placement aren't as refined as we would have liked it to be. Certain other specific challenges included AR persistence to save and load paths.

## Accomplishments that we're proud of

In less than 36 hours, we managed to familiarize ourselves with the workings of Xcode, ARKit, and Swift and created a working app that accomplished the task we set out to do.

## What we learned

We learnt the basics of a relatively new field of computer science and how it could be used to aid education and recollection capabilities. Moreover, we learnt the importance of time management and allocating specific tasks within a group.

## What's next for LociAR

Introducing additional forms of spatial representations and mapping colours to spatial objects
Traveling is exciting - planning, not so much. We thought about different ways to improve the vacation search process and found that visuals were key in selecting the perfect location. Because of this, we created TravelAR, an augmented reality app that allows you to physically step into scenes of different cities, then find flight information if you have found your ideal travel destination.

## What it does

TravelAR is an iOS travel application built using Apple's ARKit. On opening the app, there is a camera view in the room. Upon tapping, there will be an augmented reality "portal" to another city, where you physically walk inside another "room" and view a gallery of AR scenes from the city. If interested in travel information to the city, there is a pull-up information section where a user can find relevant flight details and prices.

## How we built it

We built TravelAR with Apple's iOS ARKit, a Flask server hosted with Microsoft Azure, and many APIs including the Amadeus Travel APIs, the Microsoft Bing Image Search API, and the WolframAlpha Simple API. The iOS application submits a “GET” request to our Flask server hosted in the cloud with Microsoft Azure. This Flask server takes in a city/location name, and it processes that string with many APIs to extract information—starting with the Amadeus Travel APIs. We hit the Amadeus endpoints with our location to gather information on popular attractions nearby, flight statistics, and other general travel information. We then port the “popular attractions” into the Microsoft Bing Image Search API to get a list of image urls that will be displayed in the iOS application. Furthermore, we use the WolframAlpha API to get information on the population. We combine all of the information with the AR to create a comprehensive visual display with helpful information.

## What's next for TravelAR

The future vision for TravelAR is creating 3d scenes that are almost indistinguishable from reality. Imagine stepping into the "Paris" portal and being able to view in 3d detail all things around you, the interactions of the community, and experience all of the tourist attractions - right from your home. We would also want to expand this experience by making it social. It would be great to see which of your friends have traveled to a particular location in the past and also to take inspiration from other people's travel experiences.
## Inspiration

We take our inspiration from our everyday lives. As avid travellers, we often run into places with foreign languages and need help with translations. As avid learners, we're always eager to add more words to our bank of knowledge. As children of immigrant parents, we know how difficult it is to grasp a new language and how comforting it is to hear the voice in your native tongue. LingoVision was born with these inspirations and these inspirations were born from our experiences.

## What it does

LingoVision uses AdHawk MindLink's eye-tracking glasses to capture foreign words or sentences as pictures when given a signal (double blink). Those sentences are played back in an audio translation (either using an earpiece, or out loud with a speaker) in your preferred language of choice. Additionally, LingoVision stores all of the old photos and translations for future review and study.

## How we built it

We used the AdHawk MindLink eye-tracking classes to map the user's point of view, and detect where exactly in that space they're focusing on. From there, we used Google's Cloud Vision API to perform OCR and construct bounding boxes around text. We developed a custom algorithm to infer what text the user is most likely looking at, based on the vector projected from the glasses, and the available bounding boxes from CV analysis.

After that, we pipe the text output into the DeepL translator API to a language of the users choice. Finally, the output is sent to Google's text to speech service to be delivered to the user.

We use Firebase Cloud Firestore to keep track of global settings, such as output language, and also a log of translation events for future reference.

## Challenges we ran into

* Getting the eye-tracker to be properly calibrated (it was always a bit off than our view)
* Using a Mac, when the officially supported platforms are Windows and Linux (yay virtualization!)

## Accomplishments that we're proud of

* Hearing the first audio playback of a translation was exciting
* Seeing the system work completely hands free while walking around the event venue was super cool!

## What we learned

* we learned about how to work within the limitations of the eye tracker

## What's next for LingoVision

One of the next steps in our plan for LingoVision is to develop a dictionary for individual words. Since we're all about encouraging learning, we want to our users to see definitions of individual words and add them in a dictionary.

Another goal is to eliminate the need to be tethered to a computer. Computers are the currently used due to ease of development and software constraints. If a user is able to simply use eye tracking glasses with their cell phone, usability would improve significantly.