## Inspiration 💥

Let's be honest... Presentations can be super boring to watch—*and* to present.
But, what if you could bring your biggest ideas to life in a VR world that literally puts you *in* the PowerPoint? Step beyond slides and into the future with SuperStage!

## What it does 🌟

SuperStage works in 3 simple steps:

1. Export any slideshow from PowerPoint, Google Slides, etc. as a series of images and import them into SuperStage.
2. Join your work/team/school meeting from your everyday video conferencing software (Zoom, Google Meet, etc.).
3. Instead of screen-sharing your PowerPoint window, screen-share your SuperStage window!

And just like that, your audience can watch your presentation as if you were Tim Cook in an Apple Keynote. You see a VR environment that feels exactly like standing up and presenting in real life, and the audience sees a 2-dimensional, front-row seat video of you on stage. It’s simple and only requires the presenter to own a VR headset.

Intuition was our goal when designing SuperStage: instead of using a physical laser pointer and remote, we used full-hand tracking to allow you to be the wizard that you are, pointing out content and flicking through your slides like magic. You can even use your hands to trigger special events to spice up your presentation! Make a fist with one hand to switch between 3D and 2D presenting modes, and make two thumbs-up to summon an epic fireworks display. Welcome to the next dimension of presentations!

## How we built it 🛠️

SuperStage was built using Unity 2022.3 and the C# programming language. A Meta Quest 2 headset was the hardware portion of the hack—we used the 4 external cameras on the front to capture hand movements and poses. We built our UI/UX using ray interactables in Unity to be able to flick through slides from a distance.

## Challenges we ran into 🌀

* 2-camera system. SuperStage is unique since we have to present 2 different views—one for the presenter and one for the audience. Some objects and UI in our scene must be occluded from view depending on the camera.
* Dynamic, automatic camera movement, which locked onto the player when not standing in front of a slide and balanced both slide + player when they were in front of a slide.

To build these features, we used multiple rendering layers in Unity where we could hide objects from one camera and make them visible to the other. We also wrote scripting to smoothly interpolate the camera between points and track the Quest position at all times.

## Accomplishments that we're proud of 🎊

* We’re super proud of our hand pose detection and gestures: it really feels so cool to “pull” the camera in with your hands to fullscreen your slides.
* We’re also proud of how SuperStage uses the extra dimension of VR to let you do things that aren’t possible on a laptop: showing and manipulating 3D models with your hands, and immersing the audience in a different 3D environment depending on the slide. These things add so much to the watching experience and we hope you find them cool!

## What we learned 🧠

Justin: I found learning about hand pose detection so interesting. Reading documentation and even anatomy diagrams about terms like finger abduction, opposition, etc. was like doing a science fair project.

Lily: The camera system! Learning how to run two non-conflicting cameras at the same time was super cool. The moment that we first made the switch from 3D -> 2D using a hand gesture was insane to see actually working.

Carolyn: I had a fun time learning to make cool 3D visuals!! I learned so much from building the background environment and figuring out how to create an awesome firework animation—especially because this was my first time working with Unity and C#! I also grew an even deeper appreciation for the power of caffeine… but let’s not talk about that part :)

## What's next for SuperStage ➡️

Dynamically generating presentation boards to spawn as the presenter paces the room
Providing customizable avatars to add a more personal touch to SuperStage
Adding a lip-sync feature that takes volume metrics from the Oculus headset to generate mouth animations
## 💡 Inspiration💡

Our team is saddened by the fact that so many people think that COVID-19 is obsolete when the virus is still very much relevant and impactful to us. We recognize that there are still a lot of people around the world that are quarantining—which can be a very depressing situation to be in. We wanted to create some way for people in quarantine, now or in the future, to help them stay healthy both physically and mentally; and to do so in a fun way!

## ⚙️ What it does ⚙️

We have a full-range of features. Users are welcomed by our virtual avatar, Pompy! Pompy is meant to be a virtual friend for users during quarantine. Users can view Pompy in 3D to see it with them in real-time and interact with Pompy. Users can also view a live recent data map that shows the relevance of COVID-19 even at this time. Users can also take a photo of their food to see the number of calories they eat to stay healthy during quarantine. Users can also escape their reality by entering a different landscape in 3D. Lastly, users can view a roadmap of next steps in their journey to get through their quarantine, and to speak to Pompy.

## 🏗️ How we built it 🏗️

### 🟣 Echo3D 🟣

We used Echo3D to store the 3D models we render. Each rendering of Pompy in 3D and each landscape is a different animation that our team created in a 3D rendering software, Cinema 4D. We realized that, as the app progresses, we can find difficulty in storing all the 3D models locally. By using Echo3D, we download only the 3D models that we need, thus optimizing memory and smooth runtime. We can see Echo3D being much more useful as the animations that we create increase.

### 🔴 An Augmented Metaverse in Swift 🔴

We used Swift as the main component of our app, and used it to power our Augmented Reality views (ARViewControllers), our photo views (UIPickerControllers), and our speech recognition models (AVFoundation). To bring our 3D models to Augmented Reality, we used ARKit and RealityKit in code to create entities in the 3D space, as well as listeners that allow us to interact with 3D models, like with Pompy.

### ⚫ Data, ML, and Visualizations ⚫

There are two main components of our app that use data in a meaningful way. The first and most important is using data to train ML algorithms that are able to identify a type of food from an image and to predict the number of calories of that food. We used OpenCV and TensorFlow to create the algorithms, which are called in a Python Flask server. We also used data to show a choropleth map that shows the active COVID-19 cases by region, which helps people in quarantine to see how relevant COVID-19 still is (which it is still very much so)!

## 🚩 Challenges we ran into

We wanted a way for users to communicate with Pompy through words and not just tap gestures. We planned to use voice recognition in AssemblyAI to receive the main point of the user and create a response to the user, but found a challenge when dabbling in audio files with the AssemblyAI API in Swift. Instead, we overcame this challenge by using a Swift-native Speech library, namely AVFoundation and AVAudioPlayer, to get responses to the user!

## 🥇 Accomplishments that we're proud of

We have a functioning app of an AR buddy that we have grown heavily attached to. We feel that we have created a virtual avatar that many people really can fall for while interacting with it, virtually traveling places, talking with it, and getting through quarantine happily and healthily.

## 📚 What we learned

For the last 36 hours, we learned a lot of new things from each other and how to collaborate to make a project.

## ⏳ What's next for ?

We can use Pompy to help diagnose the user’s conditions in the future; asking users questions about their symptoms and their inner thoughts which they would otherwise be uncomfortable sharing can be more easily shared with a character like Pompy. While our team has set out for Pompy to be used in a Quarantine situation, we envision many other relevant use cases where Pompy will be able to better support one's companionship in hard times for factors such as anxiety and loneliness. Furthermore, we envisage the Pompy application being a resource hub for users to improve their overall wellness. Through providing valuable sleep hygiene, exercise tips and even lifestyle advice, Pompy will be the one-stop, holistic companion for users experiencing mental health difficulties to turn to as they take their steps towards recovery.

\*\*we had to use separate github workspaces due to conflicts.
## Inspiration

We were inspired to connect people over the world in a more physical way than what the internet normally provides for us. We wanted to build a virtual reality in which all the people in our reality could see each other and interact with each other- potentially from across the globe.

## What it does

We built a dueling game in Unity in which two players may join the same game over the internet and duel in virtual reality. Using the Kinect allowed us to render virtual bodies of both players in the game, which both of the players can see. The players can fire shots at each other in a friendly duel by touching their left or right elbow with the opposite hand. All in all, we combined functionality of the Kinect with the Samsung Gear in order to create a prototype of a truly immersive VR experience.

## How we built it

We developed our game in Unity. We used Unity-Kinect plugins to gather data from the connect in the Unity framework. The Kinect sends data to the computer, which reads in each vertex of the Kinect's representation of the player's body. These vectors are transferred to the Samsung phone via Open Sound Control. The phone then creates an avatar for the player in game, with the camera centered on the player's head. The Samsung Gear controls the direction of the camera. We used the Photon plugin for Unity to create online multiplayer functionality in our game.

## Challenges we ran into

We ran into trouble using the Samsung Gear to control the camera as the Gear seems to initialize the camera direction based on the it's position at the moment the app starts. This meant that the Kinect and the Gear's ideas of the player's orientation were not always the same.

## Accomplishments that We're proud of

We are very proud of our integration of VR and Kinect; specifically, getting both Open Sound Control and Photon to integrate with Unity in order to transfer information between the Kinect, the computer, and the Gear. It was important to us that the Gear and Computer communicate locally over the local Wifi network, but that the Gear could also connect with other Gears anywhere in the world, and we succeeded in implementing these goals.

## What We learned

We definitely learned about connecting devices with Open Sound Control, and about creating multiplayer Unity games with Photon. None of us had worked on the Kinect or any VR devices prior to the Hackathon, so we had a lot of fun learning about the capabilities of these devices.

## What's next for Neo Simulator

We want to add GUI's to better facilitate connections between more than two people connecting with players of their choice. If we can aquire Gear's to continue working on, we would love to continue working on improving our game. It's already a lot of fun to play; imagine how it will look after we get some sleep!