## Inspiration

The beginnings of this idea came from long road trips. When driving having good
visibility is very important. When driving into the sun, the sun visor never seemed
to be able to actually cover the sun. When driving at night, the headlights of
oncoming cars made for a few moments of dangerous low visibility. Why isn't there
a better solution for these things? We decided to see if we could make one, and
discovered a wide range of applications for this technology, going far beyond
simply blocking light.

## What it does

EyeHUD is able to track objects on opposite sides of a transparent LCD screen in order to
render graphics on the screen relative to all of the objects it is tracking. i.e. Depending on where the observer and the object of interest are located on the each side of the screen, the location of the graphical renderings are adjusted

Our basic demonstration is based on our original goal of blocking light. When sitting
in front of the screen, eyeHUD uses facial recognition to track the position of
the users eyes. It also tracks the location of a bright flash light on the opposite side
of the screen with a second camera. It then calculates the exact position to render a dot on the screen
that completely blocks the flash light from the view of the user no matter where
the user moves their head, or where the flash light moves. By tracking both objects
in 3D space it can calculate the line that connects the two objects and then where
that line intersects the monitor to find the exact position it needs to render graphics
for the particular application.

## How we built it

We found an LCD monitor that had a broken backlight. Removing the case and the backlight
from the monitor left us with just the glass and liquid crystal part of the display.
Although this part of the monitor is not completely transparent, a bright light would
shine through it easily. Unfortunately we couldn't source a fully transparent display
but we were able to use what we had lying around. The camera on a laptop and a small webcam
gave us the ability to track objects on both sides of the screen.

On the software side we used OpenCV's haar cascade classifier in python to perform facial recognition.
Once the facial recognition is done we must locate the users eyes in their face in pixel space for
the user camera, and locate the light with the other camera in its own pixel space. We then wrote
an algorithm that was able to translate the two separate pixel spaces into real 3D space, calculate
the line that connects the object and the user, finds the intersection of this line and the monitor,
then finally translates this position into pixel space on the monitor in order to render a dot.

## Challenges we ran Into

First we needed to determine a set of equations that would allow us to translate between the three separate
pixel spaces and real space. It was important not only to be able to calculate this transformation, but
we also needed to be able to calibrate the position and the angular resolution of the cameras. This meant
that when we found our equations we needed to identify the linearly independent parts of the equation to figure
out which parameters actually needed to be calibrated.

Coming up with a calibration procedure was a bit of a challenge. There were a number of calibration parameters
that we needed to constrain by making some measurements. We eventually solved this by having the monitor render
a dot on the screen in a random position. Then the user would move their head until the dot completely blocked the
light on the far side of the monitor. We then had the computer record the positions in pixel space of all three objects.
This then told the computer that these three pixel space points correspond to a straight line in real space.
This provided one data point. We then repeated this process several times (enough to constrain all of the degrees of freedom
 in the system). After we had a number of data points we performed a chi-squared fit to the line defined by these points
 in the multidimensional calibration space. The parameters of the best fit line determined our calibration parameters to use
in the transformation algorithm.

This calibration procedure took us a while to perfect but we were very happy with the speed and accuracy we were able to calibrate at.

Another difficulty was getting accurate tracking on the bright light on the far side of the monitor. The web cam we were
using was cheap and we had almost no access to the settings like aperture and exposure which made it so the light would
easily saturate the CCD in the camera. Because the light was saturating and the camera was trying to adjust its own exposure,
other lights in the room were also saturating the CCD and so even bright spots on the white walls were being tracked as well.
We eventually solved this problem by reusing the radial diffuser that was on the backlight of the monitor we took apart.
This made any bright spots on the walls diffused well under the threshold for tracking. Even after this we had a bit of trouble
locating the exact center of the light as we were still getting a bit of glare from the light on the camera lens. We were
able to solve this problem by applying a gaussian convolution to the raw video before trying any tracking. This allowed us
to accurately locate the center of the light.

## Accomplishments that we are proud of

The fact that our tracking display worked at all we felt was a huge accomplishments. Every stage of this project felt like a
huge victory. We started with a broken LCD monitor and two white boards full of math. Reaching a well working final product
was extremely exciting for all of us.

## What we learned

None of our group had any experience with facial recognition or the OpenCV library. This was a great opportunity to dig into
a part of machine learning that we had not used before and build something fun with it.

## What's next for eyeHUD

Expanding the scope of applicability.

* Infrared detection for pedestrians and wildlife in night time conditions
* Displaying information on objects of interest
* Police information via license plate recognition

Transition to a fully transparent display and more sophisticated cameras.

General optimization of software.
## Inspiration

Knowtworthy is a startup that all three of us founded together, with the mission to make meetings awesome. We have spent this past summer at the University of Toronto’s Entrepreneurship Hatchery’s incubator executing on our vision. We’ve built a sweet platform that solves many of the issues surrounding meetings but we wanted a glimpse of the future: entirely automated meetings. So we decided to challenge ourselves and create something that the world has never seen before: sentiment analysis for meetings while transcribing and attributing all speech.

## What it does

While we focused on meetings specifically, as we built the software we realized that the applications for real-time sentiment analysis are far more varied than initially anticipated. Voice transcription and diarisation are very powerful for keeping track of what happened during a meeting but sentiment can be used anywhere from the boardroom to the classroom to a psychologist’s office.

## How I built it

We felt a web app was best suited for software like this so that it can be accessible to anyone at any time. We built the frontend on React leveraging Material UI, React-Motion, Socket IO and ChartJS. The backed was built on Node (with Express) as well as python for some computational tasks. We used GRPC, Docker and Kubernetes to launch the software, making it scalable right out of the box.

For all relevant processing, we used Google Speech-to-text, Google Diarization, Stanford Empath, SKLearn and Glove (for word-to-vec).

## Challenges I ran into

Integrating so many moving parts into one cohesive platform was a challenge to keep organized but we used trello to stay on track throughout the 36 hours.
Audio encoding was also quite challenging as we ran up against some limitations of javascript while trying to stream audio in the correct and acceptable format.
Apart from that, we didn’t encounter any major roadblocks but we were each working for almost the entire 36-hour stretch as there were a lot of features to implement.

## Accomplishments that I'm proud of

We are super proud of the fact that we were able to pull it off as we knew this was a challenging task to start and we ran into some unexpected roadblocks. There is nothing else like this software currently on the market so being first is always awesome.

## What I learned

We learned a whole lot about integration both on the frontend and the backend. We prototyped before coding, introduced animations to improve user experience, too much about computer store numbers (:p) and doing a whole lot of stuff all in real time.

## What's next for Knowtworthy Sentiment

Knowtworthy Sentiment aligns well with our startup’s vision for the future of meetings so we will continue to develop it and make it more robust before integrating it directly into our existing software. If you want to check out our stuff you can do so here: <https://knowtworthy.com/>
## Inspiration

I came up with these idea from two thoughts:

(1) We live in a world in which sensors are everywhere and on everything. In this world, coming up with an embedded system isn't as hard as it used to be. Rather, now the issue is what kind of sensor and how accurate the sensor is to retrieve the data from.

(2) Humans have five major senses: touch, smell, sight, sound, and taste. How much do we know about these? Do the computers know what we are feeling in terms of these senses? The answer is that we only know what we are listening to (a.k.a. Shazam). **Why can't we do this for vision?**

The closest platform I know of that indirectly solves this problem is Google Glass. However, how do you tell what you are exactly looking at when there are multiple objects within the camera capture? This is where I got the idea for this project; I want to specify where your sight lies on the Google Glass-like camera capture and identify what that exact object is.

## What it does

As I briefly mentioned above, it reads the electrical potential difference across your eyes, horizontally and vertically. Then, it maps these readings into a 2D coordinate on the picture that is taken from the camera that lies on pair of glasses. This represents what you are looking at in that picture.

On this version, I identify all known objects within the picture, get the boundaries, and find the closest object boundary to the sight-point. That is the final output of this platform; the predicted object that you are looking at.

## How I built it

I started off with the hardware. I knew that the Electroocculogram signals have few mV and 1~30Hz signal range. Therefore, I put an instrumentation amplifier and few high pass and low pass filters to get rid of noise.

This voltage output is then fed into Arduino for ADC, from which I can get the digital serial data.

Finally, I used OpenCV 3.0.0 library to do basic computer vision process. I register few objects that are relatively easy to recognize, and I spot all of them within the camera capture. SIFT and FLANN was used for this purpose. I find the closest object boundary from this list of detected objects.

## Challenges I ran into

There were a LOT of challenges due to this project's cross-hardware-software nature:

**(1) Hardware debugging.** I fried two op-amps and one instrumentation amplifier because I fed in wrong voltage into them. I also had to play around with resistor values and add few filters to get useable data.

**(2) Arduino and PySerial interface.** Controlling serial data flow from Python's end was quite difficult. Lots of errors popping up, and had to debug a lot.

**(3) OpenCV 3.0.0.** This newest version of OpenCV is notoriously hard to install. It really should not be. I lost precious team members because OpenCV 3.0.0 refused to work with Windows 10.

**(4) Noisy Signal.** Body signals are noisy by nature. Coming up with a scheme that will work well with this noisy data was quite hard.

**(5) Solo project (unintentionally).** As stated from point above, OpenCV 3.0.0 took my team members away, and I had to come up with the entire project myself.

**(5+) Lack of sleep.** It's hard.

## Accomplishments that I'm proud of

I am proud that I was able to complete this project with little time and little manpower. Project could have been a lot more sophisticated otherwise, but I am very satisfied that I was able to pull all the way from hardware data collection onto computer vision identification to decently functional level.

## What I learned

(1) For projects dealing with unknown concepts, it is not only important for you to be knowledgeable with the material but also your teammates. Keep them updated and prepared as much as you are.

(2) Body signal is probably the next wave of sensors that should be mastered, and I think there is a lot of potential there. Even with this time limited, undermanned project, I was able to get somewhat of a working product which I think can be used for a lot of different things. Namely, Google Glass, Microsoft Hololense, and Oculus VR.

## What's next for Oqulo

As mentioned above, more noise reduction on the hardware side, better runtime optimization of the Python code, and using external object identification API are on the table. Object identification is no new subject and there are people with efficient algorithms and large object image library sitting in their database. Should be very easy to improve object identification performance.

Also, "wet" electrodes are very uncomfortable to keep them on for 2 days straight. Dry electrodes that gently sit on the face will definitely be better.