## Inspiration

We were looking at the Apple Magic TrackPads last week since they seemed pretty cool. But then we saw the price tag, $130! That's crazy! So we set out to create a college student budget friendly "magic" trackpad.

## What it does

Papyr is a trackpad for your computer that is just a single sheet of paper, no wires, strings, or pressure detecting devices attached. Paypr allows you to browse the computer just like any other trackpad and supports clicking and scrolling.

## How we built it

We use a webcam and a whole lot of computer vision to make the magic happen. The webcam first calibrates itself by detecting the four corners of the paper and maps every point on the sheet to a location on the actual screen. Our program then tracks the finger on the sheet by analyzing the video stream in real time, frame by frame, blurring, thresholding, performing canny edge detection, then detecting the contours in the final result. The furthest point in the hand’s contour corresponds to the user's fingertip and is translated into both movement and actions on the computer screen. Clicking is detailed in the next section, with scrolling is activated by double clicking.

## Challenges we ran into

Light sensitivity proved to be very challenging since depending on the environment, the webcam would sometimes have trouble tracking our fingers. However, finding a way to detect clicking was by far the most difficult part of the project. The problem is the webcam has no sense of depth perception: it sees each frame as a 2D image and as a result there is no way to detect if your hand is on or off the paper. We turned to the Internet hoping for some previous work that would guide us in the right direction, but everything we found required either glass panels, infrared sensors, or other non college student budget friendly hardware. We were on our own. We made many attempts including: having the user press down very hard on the paper so that their skin would turn white and detect this change of color, track the shadow the user's finger makes on the paper and detect when the shadow disappears, which occurs when the user places his finger on the paper.
None of these methods proved fruitful, so we sat down and for the better part of 5 hours thought about how to solve this issue. Finally, what worked for us was to track the “hand pixel” changes across several frames to detect a valid sequence that can qualify as a “click”. Given the 2D image perception with our web cam, it was no easy task and there was a lot of experimentation that went into this.

## Accomplishments that we're proud of

We are extremely proud of getting clicking to work. It was no easy feat. We also developed our own algorithms for fingertip tracking and click detection and wrote code from scratch. We set out to create a cheap trackpad and we were able to. In the end we transformed a piece of paper, something that is portable and available nearly anywhere, into a makeshift-high tech device with only the help of a standard webcam. Also one of the team members was able to win a ranked game of Hearthstone using a piece of paper so that was cool (not the match shown in the video).

## What we learned

From normalizing the environment's lighting and getting rid of surrounding noise to coming up with the algorithm to provide depth perception to a 2D camera, this project taught us a great deal about computer vision. We also learned about efficiency and scalability since numerous calculations need to be made each second in analyze each frame and everything going on in them.

## What's next for Papyr - A Paper TrackPad

We would like to improve the accuracy and stability of Papyr. This would allow Papyr to function as a very cheap replacement for Wacom digital tablets. Papyr already supports various "pointers" such as fingers or pens.
## Inspiration

The beginnings of this idea came from long road trips. When driving having good
visibility is very important. When driving into the sun, the sun visor never seemed
to be able to actually cover the sun. When driving at night, the headlights of
oncoming cars made for a few moments of dangerous low visibility. Why isn't there
a better solution for these things? We decided to see if we could make one, and
discovered a wide range of applications for this technology, going far beyond
simply blocking light.

## What it does

EyeHUD is able to track objects on opposite sides of a transparent LCD screen in order to
render graphics on the screen relative to all of the objects it is tracking. i.e. Depending on where the observer and the object of interest are located on the each side of the screen, the location of the graphical renderings are adjusted

Our basic demonstration is based on our original goal of blocking light. When sitting
in front of the screen, eyeHUD uses facial recognition to track the position of
the users eyes. It also tracks the location of a bright flash light on the opposite side
of the screen with a second camera. It then calculates the exact position to render a dot on the screen
that completely blocks the flash light from the view of the user no matter where
the user moves their head, or where the flash light moves. By tracking both objects
in 3D space it can calculate the line that connects the two objects and then where
that line intersects the monitor to find the exact position it needs to render graphics
for the particular application.

## How we built it

We found an LCD monitor that had a broken backlight. Removing the case and the backlight
from the monitor left us with just the glass and liquid crystal part of the display.
Although this part of the monitor is not completely transparent, a bright light would
shine through it easily. Unfortunately we couldn't source a fully transparent display
but we were able to use what we had lying around. The camera on a laptop and a small webcam
gave us the ability to track objects on both sides of the screen.

On the software side we used OpenCV's haar cascade classifier in python to perform facial recognition.
Once the facial recognition is done we must locate the users eyes in their face in pixel space for
the user camera, and locate the light with the other camera in its own pixel space. We then wrote
an algorithm that was able to translate the two separate pixel spaces into real 3D space, calculate
the line that connects the object and the user, finds the intersection of this line and the monitor,
then finally translates this position into pixel space on the monitor in order to render a dot.

## Challenges we ran Into

First we needed to determine a set of equations that would allow us to translate between the three separate
pixel spaces and real space. It was important not only to be able to calculate this transformation, but
we also needed to be able to calibrate the position and the angular resolution of the cameras. This meant
that when we found our equations we needed to identify the linearly independent parts of the equation to figure
out which parameters actually needed to be calibrated.

Coming up with a calibration procedure was a bit of a challenge. There were a number of calibration parameters
that we needed to constrain by making some measurements. We eventually solved this by having the monitor render
a dot on the screen in a random position. Then the user would move their head until the dot completely blocked the
light on the far side of the monitor. We then had the computer record the positions in pixel space of all three objects.
This then told the computer that these three pixel space points correspond to a straight line in real space.
This provided one data point. We then repeated this process several times (enough to constrain all of the degrees of freedom
 in the system). After we had a number of data points we performed a chi-squared fit to the line defined by these points
 in the multidimensional calibration space. The parameters of the best fit line determined our calibration parameters to use
in the transformation algorithm.

This calibration procedure took us a while to perfect but we were very happy with the speed and accuracy we were able to calibrate at.

Another difficulty was getting accurate tracking on the bright light on the far side of the monitor. The web cam we were
using was cheap and we had almost no access to the settings like aperture and exposure which made it so the light would
easily saturate the CCD in the camera. Because the light was saturating and the camera was trying to adjust its own exposure,
other lights in the room were also saturating the CCD and so even bright spots on the white walls were being tracked as well.
We eventually solved this problem by reusing the radial diffuser that was on the backlight of the monitor we took apart.
This made any bright spots on the walls diffused well under the threshold for tracking. Even after this we had a bit of trouble
locating the exact center of the light as we were still getting a bit of glare from the light on the camera lens. We were
able to solve this problem by applying a gaussian convolution to the raw video before trying any tracking. This allowed us
to accurately locate the center of the light.

## Accomplishments that we are proud of

The fact that our tracking display worked at all we felt was a huge accomplishments. Every stage of this project felt like a
huge victory. We started with a broken LCD monitor and two white boards full of math. Reaching a well working final product
was extremely exciting for all of us.

## What we learned

None of our group had any experience with facial recognition or the OpenCV library. This was a great opportunity to dig into
a part of machine learning that we had not used before and build something fun with it.

## What's next for eyeHUD

Expanding the scope of applicability.

* Infrared detection for pedestrians and wildlife in night time conditions
* Displaying information on objects of interest
* Police information via license plate recognition

Transition to a fully transparent display and more sophisticated cameras.

General optimization of software.
## Inspiration

I was inspired to make this device while sitting in physics class. I really felt compelled to make something that I learned inside the classroom and apply my education to something practical. Growing up I always remembered playing with magnetic kits and loved the feeling of repulsion between magnets.

## What it does

There is a base layer of small magnets all taped together so the North pole is facing up. There are hall effect devices to measure the variances in magnetic field that is created by the user's magnet attached to their finger. This allows the device to track the user's finger and determine how they are interacting with the magnetic field pointing up.

## How I built it

It is build using the intel edison. Each hall effect device is either on or off depending if there is a magnetic field pointing down through the face of the black plate. This determines where the user's finger is. From there the analog data is sent via serial port to the processing program on the computer that demonstrates that it works. That just takes the data and maps the motion of the object.

## Challenges I ran into

There are many challenges I faced. Two of them dealt with just the hardware. I bought the wrong type of sensors. These are threshold sensors which means they are either on or off instead of linear sensors that give a voltage proportional to the strength of magnetic field around it. This would allow the device to be more accurate. The other one deals with having alot of very small worn out magnets. I had to find a way to tape and hold them all together because they are in an unstable configuration to create an almost uniform magnetic field on the base. Another problem I ran into was dealing with the edison, I was planning on just controlling the mouse to show that it works but the mouse library only works with the arduino leonardo. I had to come up with a way to transfer the data to another program, which is how i came up dealing with serial ports and initially tried mapping it into a Unity game.

## Accomplishments that I'm proud of

I am proud of creating a hardware hack that I believe is practical. I used this device as a way to prove the concept of creating a more interactive environment for the user with a sense of touch rather than things like the kinect and leap motion that track your motion but it is just in thin air without any real interaction. Some areas this concept can be useful in is in learning environment or helping people in physical therapy learning to do things again after a tragedy, since it is always better to learn with a sense of touch.

## What I learned

I had a grand vision of this project from thinking about it before hand and I thought it was going to theoretically work out great! I learned how to adapt to many changes and overcoming them with limited time and resources. I also learned alot about dealing with serial data and how the intel edison works on a machine level.

## What's next for Tactile Leap Motion

Creating a better prototype with better hardware(stronger magnets and more accurate sensors)