## Inspiration

As AR becomes develops and becomes more discrete, we are going to see it more and more in every day life. We were excited by the social side of AR, and how integrating social networks in real life interactions brings the "social" side of social networking back. We wanted to find a way to identify humans through facial tracking/recognition and bring up a known set of social characteristics to augment your interaction.

## What it does

Once you've talked to a person for a certain amount of time, SocialEyes recognizes and stores their face on the Android app. It groups those faces by person and you then have the ability to connect their face to their Facebook account. From then on, whenever you encounter that person, SocialEyes tracks and recognizes the face and brings up his or her information in a HUD environment. SocialEyes can even tell you the persons heart rate as you are talking to them.

## How I built it

We tuned Haar cascades trained for finding faces in OpenCV to locate the positions of heads in space from the view of the camera of our Meta Augmented Reality (AR) Headset. If we've never seen the face before, we send the face to our back end, which can be accessed and edited using our companion Android app. From here, faces are sent to Azure, where they are grouped using Project Oxford. The groups can be tagged using our Android app, and then linked to Facebook using the Facebook API. The next time that we see the face, we can send the face to our back end, which then sends the face to Azure for identification (among the different tagged face groups we've accumulated). The back end transmits information about the person originally grabbed from the Facebook API back to the computer running the headset, which then displays the information on the AR headset next to the identified person's head.

At the same time, we also calculated heart rates of subjects given only the AR headset's video feed using eulerian magnification, a technique originally developed in MIT's Computer Science and Artificial intelligence laboratory. We captured groups of pixels located on people's foreheads located using a combination of Haar cascades for the eyes and head. We then manipulated these captured groups of pixels using a signal processing library (part of OpenMDAO) for calculation of the subject's heart rates (after a small calibration period).

## Challenges I ran into

The scope of this project presented the most issues--we were working on basically three different project that all had to come together and function smoothly. Latency was easily the largest issue we had that covered all three projects. We had to use computer vision to detect and track heads in the frame, while simultaneously updating the UI and sending off our images for external processing in Azure. Concurrently, we had to find the heart rate through image processing. We were also working with a Meta kit that is still only for developers, so there are limitations on things like field of view and resolution that we had to work around. The facial tracking had

## Accomplishments that I'm proud of

We're most proud of the sheer complexity of this project. There was so much to do on so many different platforms that we couldn't be sure anything would work together smoothly. Because it was so multifaceted, we had to work evenly as a team and strategically divide up tasks, so we're also proud of how well the team worked together.

## What I learned

We learned a lot about Azure and Facebook Graph API, which were both instrumental in our project's success. We had to learn a lot in a very short amount of time, but both ended up working flawlessly with our product.

## What's next for SocialEyes AR

What's most exciting about this project is the interaction between AR and humans. AR headsets can only augment reality so much without interacting with humans, and we think this is the next step in human-to-tech interaction. Five or ten years down the road, this is the kind of thing that will humanize Artificial Intelligence--the ability to identify and "know" a human being by face.
## Inspiration

There are approximately **10 million** Americans who suffer from visual impairment, and over **5 million Americans** suffer from Alzheimer's dementia. This weekend our team decided to help those who were not as fortunate. We wanted to utilize technology to create a positive impact on their quality of life.

## What it does

We utilized a smartphone camera to analyze the surrounding and warn visually impaired people about obstacles that were in their way. Additionally, we took it a step further and used the **Azure Face API** to detect the faces of people that the user interacted with and we stored their name and facial attributes that can be recalled later. An Alzheimer's patient can utilize the face recognition software to remind the patient of who the person is, and when they last saw him.

## How we built it

We built our app around **Azure's APIs**, we created a **Custom Vision** network that identified different objects and learned from the data that we collected from the hacking space. The UI of the iOS app was created to be simple and useful for the visually impaired, so that they could operate it without having to look at it.

## Challenges we ran into

Through the process of coding and developing our idea we ran into several technical difficulties. Our first challenge was to design a simple UI, so that the visually impaired people could effectively use it without getting confused. The next challenge we ran into was attempting to grab visual feed from the camera, and running them fast enough through the Azure services to get a quick response. Another challenging task that we had to accomplish was to create and train our own neural network with relevant data.

## Accomplishments that we're proud of

We are proud of several accomplishments throughout our app. First, we are especially proud of setting up a clean UI with two gestures, and voice control with speech recognition for the visually impaired. Additionally, we are proud of having set up our own neural network, that was capable of identifying faces and objects.

## What we learned

We learned how to implement **Azure Custom Vision and Azure Face APIs** into **iOS**, and we learned how to use a live camera feed to grab frames and analyze them. Additionally, not all of us had worked with a neural network before, making it interesting for the rest of us to learn about neural networking.

## What's next for BlindSpot

In the future, we want to make the app hands-free for the visually impaired, by developing it for headsets like the Microsoft HoloLens, Google Glass, or any other wearable camera device.
## Inspiration

When networking / talking with people, names and other info is typically forgotten very quickly. Having information readily available about the potential employers while you speak can greatly improve your networking experience.

## What it does

Uses facial recognition to determine locations of faces in a live video feed and overlays info about detected people. This information is viewed in real-time on a VR display as an augmented reality pass-through. The system allows users to have the most relevant info in their sight while talking to others, reducing stress and increasing efficiency in communication.

## How we built it

The frontend is a Unity 3D Android app using ARFoundation for base AR functionality. Facial detection is done by sending a JPG every 10 frames to the Azure Face API backend which returns bounding boxes of all faces. Next the bounding box centers are mapped into the 3D world to display info text around faces.

## Challenges we ran into

The original tools we used (ARCore) did not allow for facial detection using a back camera (by design). We had to change the tools we used and develop our own solutions for certain tasks.
Mapping screen space coordinates from camera to world coordinates was difficult to do reliably.
We had access to limited hardware, mainly only having a VR headset instead of an AR one.

## Accomplishments that we are proud of

* Integrated many different tools to make application work
* Able to recognize faces, do augmented reality and display visual information in a suitable way for a VR headset.

## What we learned

* Azure Face API
* Coding Unity 3D games in C#
* Google Cloud Platform
* HTTP requests

## What's next for NetworkSmartAR

Adding more displayed information about individuals. Adding voice recognition for names and conversation details being added to displayed information.