## 💡Inspiration

Gaming is often associated with sitting for long periods of time in front of a computer screen, which can have negative physical effects. In recent years, consoles such as the Kinect and Wii have been created to encourage physical fitness through games such as "Just Dance". However, these consoles are simply incompatible with many of the computer and arcade games that we love and cherish.

## ❓What it does

We came up with Motional at HackTheValley wanting to create a technological solution that pushes the boundaries of what we’re used to and what we can expect. Our product, Motional, delivers on that by introducing a new, cost-efficient, and platform-agnostic solution to universally interact with video games through motion capture, and reimagining the gaming experience.

Using state-of-the-art machine learning models, Motional can detect over 500 features on the human body (468 facial features, 21 hand features, and 33 body features) and use these features as control inputs to any video game.

Motional operates in 3 modes: using hand gestures, face gestures, or full-body gestures. We ship certain games out-of-the-box such as Flappy Bird and Snake, with predefined gesture-to-key mappings, so you can play the game directly with the click of a button. For many of these games, jumping in real-life (body gesture) /opening the mouth (face gesture) will be mapped to pressing the "space-bar"/"up" button.

However, the true power of Motional comes with customization. Every simple possible pose can be trained and clustered to provide a custom command. Motional will also play a role in creating a more inclusive gaming space for people with accessibility needs, who might not physically be able to operate a keyboard dexterously.

## 🤔 How we built it

First, a camera feed is taken through Python OpenCV. We then use Google's Mediapipe models to estimate the positions of the features of our subject. To learn a new gesture, we first take a capture of the gesture and store its feature coordinates generated by Mediapipe. Then, for future poses, we compute a similarity score using euclidean distances. If this score is below a certain threshold, we conclude that this gesture is the one we trained on. An annotated image is generated as an output through OpenCV. The actual keyboard presses are done using PyAutoGUI.

We used Tkinter to create a graphical user interface (GUI) where users can switch between different gesture modes, as well as select from our current offering of games. We used MongoDB as our database to keep track of scores and make a universal leaderboard.

## 👨‍🏫 Challenges we ran into

Our team didn't have much experience with any of the stack before, so it was a big learning curve. Two of us didn't have a lot of experience in Python. We ran into many dependencies issues, and package import errors, which took a lot of time to resolve. When we initially were trying to set up MongoDB, we also kept timing out for weird reasons. But the biggest challenge was probably trying to write code while running on 2 hours of sleep...

## 🏆 Accomplishments that we're proud of

We are very proud to have been able to execute our original idea from start to finish. We managed to actually play games through motion capture, both with our faces, our bodies, and our hands. We were able to store new gestures, and these gestures were detected with very high precision and low recall after careful hyperparameter tuning.

## 📝 What we learned

We learned a lot, both from a technical and non-technical perspective. From a technical perspective, we learned a lot about the tech stack (Python + MongoDB + working with Machine Learning models). From a non-technical perspective, we worked a lot working together as a team and divided up tasks!

## ⏩ What's next for Motional

We would like to implement a better GUI for our application and release it for a small subscription fee as we believe there is a market for people that would be willing to invest money into an application that will help them automate and speed up everyday tasks while providing the ability to play any game they want the way they would like. Furthermore, this could be an interesting niche market to help gamify muscle rehabilition, especially for children.
## Inspiration

MISSION: Our mission is to create an intuitive and precisely controlled arm for situations that are tough or dangerous for humans to be in.

VISION: This robotic arm application can be used in the medical industry, disaster relief, and toxic environments.

## What it does

The arm imitates the user in a remote destination. The 6DOF range of motion allows the hardware to behave close to a human arm. This would be ideal in environments where human life would be in danger if physically present.

The HelpingHand can be used with a variety of applications, with our simple design the arm can be easily mounted on a wall or a rover. With the simple controls any user will find using the HelpingHand easy and intuitive. Our high speed video camera will allow the user to see the arm and its environment so users can remotely control our hand.

## How I built it

The arm is controlled using a PWM Servo arduino library. The arduino code receives control instructions via serial from the python script. The python script is using opencv to track the user's actions. An additional feature use Intel Realsense and Tensorflow to detect and track user's hand. It uses the depth camera to locate the hand, and use CNN to identity the gesture of the hand out of the 10 types trained. It gave the robotic arm an additional dimension and gave it a more realistic feeling to it.

## Challenges I ran into

The main challenge was working with all 6 degrees of freedom on the arm without tangling it. This being a POC, we simplified the problem to 3DOF, allowing for yaw, pitch and gripper control only. Also, learning the realsense SDK and also processing depth image was an unique experiences, thanks to the hardware provided by Dr. Putz at the nwhacks.

## Accomplishments that I'm proud of

This POC project has scope in a majority of applications. Finishing a working project within the given time frame, that involves software and hardware debugging is a major accomplishment.

## What I learned

We learned about doing hardware hacks at a hackathon. We learned how to control servo motors and serial communication. We learned how to use camera vision efficiently. We learned how to write modular functions for easy integration.

## What's next for The Helping Hand

Improve control on the arm to imitate smooth human arm movements, incorporate the remaining 3 dof and custom build for specific applications, for example, high torque motors would be necessary for heavy lifting applications.
## Inspiration

The release of Google Cardboard in 2014 launched a movement of using inexpensive materials to create modules that complement existing, commonly available devices such as smartphones, to create revolutionary experiences for users. Thus, in the same vein, we decided to bring retro to the new age using up-cycled materials found in everyday life and an Android phone. This new jukebox will change the way you listen to and interact with music on your phone.

## What it does

There are two main components of this project: (1) the interpretation of human interaction via computer vision, and (2) visualization of the selected tunes. We use the front camera of a Nexus 5 phone to determine whether the user is turning the knobs (aka cups) by looking at the movement of markers drawn on the cup. The volume is then adjusted accordingly. The music is also represented visually through a disco ball. The disco ball uses the phone's screen as a light source and thus has the potential to show different patterns and colors.

## How we built it

We used a cardboard box as the frame of the project. A Styrofoam cup was used as the volume knob, with a smartphone tracking markers along the cup's rim. The phone also serves as the source of music.

A disco ball also hangs from the top of the jukebox. It was made out of aluminum foil taped to a ping pong ball, and the glow of the phone's screen from below is enough to light the ball up.

## Challenges we ran into

Lack of materials

## Accomplishments that we're proud of

Building a cohesive product with both input (volume knob) and output (lit disco ball, music) channels.

## What we learned

* The challenge of lacking materials can make even seemingly simple ideas very difficult.
* Smartphones can be incredibly powerful in both sensing and actuation.

## What's next for CardBox JukeBoard

* Finding ways to amplify the phone's speakers using passive elements.
* Finding ways to concentrate the phone's screen light to make the disco ball even brighter.
* Making it easier to assemble and releasing it to the community.