## Inspiration

Currently, Zoom only offers live closed captioning when a human transcriber manually transcribes a meeting. We believe that users would benefit greatly from closed captions in *every* meeting, so we created Cloud Caption.

## What it does

Cloud Caption receives live system audio from a Zoom meeting or other video conference platform and translate this audio in real time to closed captioning that is displayed in a floating window. This window can be positioned on top of the Zoom meeting and it is translucent, so it will never get in the way.

## How we built it

Cloud Caption uses the Google Cloud Speech-to-Text API to automatically transcribe the audio streamed from Zoom or another video conferencing app.

## Challenges we ran into

We went through a few iterations before we were able to get Cloud Caption working. First, we started with a browser-based app that would embed Zoom, but we discovered that the Google Cloud framework isn't compatible in browser-based environments. We then pivoted to an Electron-based desktop app, but the experimental web APIs that we needed did not work. Finally, we implemented a Python-based desktop app that uses a third-party program like [Loopback](https://rogueamoeba.com/loopback/) to route the audio.

## Accomplishments that we're proud of

We are proud of our ability to think and adapt quickly and collaborate efficiently during this remote event. We're also proud that our app is a genuinely useful accessibility tool for anyone who is deaf or hard-of-hearing, encouraging all students and learners to collaborate in real time despite any personal challenges they may face. Cloud Caption is also useful for students who aren't auditory learners and prefer to learn information by reading.

Finally, we're proud of the relative ease-of-use of the application. Users only need to have Loopback (or another audio-routing program) installed on their computer in order to receive real time video speech-to-text transcription, instead of being forced to wait and re-watch a video conference later with closed captioning embedded.

## What we learned

Our team learned that specifying, controlling, and linking audio input and output sources can be an incredibly difficult task with poor support from browser and framework vendors. We also came to appreciate the values of building with accessibility as a major goal throughout the design and development process. Accessibility can often be overlooked in applications and projects of every size, so all of us have learned to prioritize developing with inclusivity in mind for our projects moving forward.

## What's next for Cloud Caption

Our next step is to integrate audio routing so that users won't need a third-party program. We would also like to explore further applications of our closed captioning application in other business or corporate uses cases for HR or training purposes, especially targeting those users who may be deaf or hard-of-hearing.
## Inspiration

as we have spent some time in our new Covid-19 world, we have noticed work from home and such has resulted in an increase of phishing and fraudulent emails. We noticed this is strongly apparent in our educational community as well noticing that our inboxes and our peers' inboxes get spammed with fraudulent E-mails that pose a risk and threat to our finances.

## What it does

MailSafe allows you to verify the integrity and legitimacy of your emails by generating a QR Code in a Google Chrome Extension pop-up for your phone to scan. The QR Code is scanned by the phone in the MailSafe Mobile App that cross references the QR Code to a list of verified email senders. If present on the list, it means the email is verified and the user can trust the email they received. If it is not on the list, the email is most likely fraudulent as it does not belong to any of your financial service providers.

## How we built it

We had used standard Google Chrome Extension structure. We constructed a main manifest file for the Chrome Extension to function and call a popup.html file upon an Chrome Extension Action that showed a button. Clicking that button generated a QR Code from a standard qrcode.js library that your phone could actually scan. Then we created the way the app would function on your mobile device using Figma.

## Challenges we ran into

The main challenge we ran into was in the Google Chrome Extension. We had to use the email sender ID to generate a QR Code. After some trial and error, and countless stackoverflow guides, we came to a library, qrcode.js, that allowed us to pass the email sender ID and generate a QR Code from it. That QR Code is functional and usable by most devices.

## Accomplishments that we're proud of

We are proud of the functional Google Chrome Extension that generates actual QR Codes for devices to scan. All in all, we are proud to have come up with an idea to protect users from fraudulent E-mails and phishing scams and proud that we have implemented a major element of the system. Now that we have that foundation, we are on our way to complete the mobile application element to have a functional system.

## What we learned

The project was a great learning experience for us at MailSafe. Spending most of our time implementing the Google Chrome Extension we learned to manipulate an existing HTML file, parse it and extract information we specifically needed such as the email sender ID. Using that we learned how to convert it into a QR Code that actually is recognized by anyone's cellular device. All in all, it was an amazing opportunity and learning experience.

## What's next for MailSafe

Our goals for MailSafe is to implement a functional phone application. Furthermore, we would like to implement the the extension add-on for non-chromium based browsers such as Mozilla Firefox. In the future, we are looking forward to wide-spread adoption by many companies, not just Bank Institutions, who would like to provide their users with a seamless and safe experience in communicating over E-mail.
## Inspiration

-Inspired by baracksdubs and many "Trump Sings" channels on YouTube, each of which invests a lot of time into manually tracking down words.
-Fully automating the process allows us to mass-produce humorous content and "fake news", bringing awareness to the ease with which modern technology allows for the production and perpetuation of generated content.
-Soon-to-emerge technologies like Adobe VOCO are poised to allow people to edit audio and human speech as seamlessly as we are currently able to edit still images.
-The inspirational lectures of Professor David J. Malan.

## What it does

We train each available "voice" by inputting a series of YouTube URL's to `main.py`.
`download.py` downloads and converts these videos to `.wav` files for use in `speech.py`, which uses Google's Cloud Speech API to create a dictionary of mappings between words and video time-stamps.
`application.py` implements user interaction: given a voice/text input via Facebook, we use these mappings to concatenate the video clips corresponding to each word.

## How we built it

First we decided on Python due to its huge speech recognition community. This also allowed us to utilize a collaborative online workspace through Cloud9 which helped facilitate concurrent collaboration.

We used google's speech api because we saw that it was very popular and supported time stamps for individual words. Also, they had very elegant json output, which was a definite bonus.

Next, we figured out how to use the packages pytube and ffmpy to grab video streams from youtube and convert them, with speed and without loss of quality, to the needed .wav and .mp4 formats.

At the same time, one of our team members learned how to use python packages to concatenate and split .mp4 videos, and built functions with which we were able to manipulate small video files with
high precision.

Following some initial successes with google speech api and mp4 manipulation, we began exploring the facebook graph api. There quite a bit of struggle here with permissions issues because many of the functions
we were trying to call were limited by permissions, and those permissions had to be granted by facebook people after review. However, we did eventually get facebook to integrate with our program.

The final step we took was to few remaining unconnected pieces of the project together and troubleshoot any issues that came up.

During the process, we were also investigating a few moonshot-type upgrades. These included ideas like the use of a sound spectrogram to find individual phonemes of words, so we could finely tune individual words, or generate
new words that were never previously said by the person.

## Challenges we ran into

A big challenge we ran into was that the Google Speech API was not extremely accurate when identifying single words. We tried various things like different file/compression types, boosting sound (normalizing/processing waveform),
improving sound quality (bitrate, sampling frequency).

Another big challenge we ran into was that when we tried splicing the small (under 1 or 2 second) video files together, we realized they lost their video component, due to issues
with key frames, negative timestamps, and video interpolation. Apparently, in order to save space, videos store key frames and interpolate between the key frames to generate the frames
in between. This is good enough to fool the human eye, but it required that we do a lot of extra work to get the correct output.

A third big challenge we ran into was that when we communicated with the facebook api through our flask website, facebook would resend our flask page post requests before we were completed
with processing the information from the previous post request. To solve this issue, we grabbed the post request information and opened new threads in python to process them in parallel.

A fourth big challenge we ran into was that wifi was so slow that it would take around 1 minute to upload a 1 minute video to Google's cloud for speech processing. Thus, in order to analyze
large videos (1+ hours) we developed a way to use multiple threads to split the video into smaller segments without destroying words and upload those segments in parallel.

## Accomplishments that we're proud of

We have a scalable, modular structure which makes future expansion easy. This allows us to easily switch APIs for each function.

## What we learned

[Web Services APIs]

>
> Speech to Text Conversion:
> --Google Cloud API
> --CMU Sphynx (Experimental Offline Speech-To-Text Processing with the English Lanugage Model)
> Facebook API Integration:
> --Accepting input from user via automated messenger bot development
> --Posting to Facebook Page
>
>
>

[Web Services Deployment]

>
> Flask and Python Interfacing
>
>
>

[Python]

>
> Multi-file Python package integration
> Team-based Development
>
>
>

[Video and Audio Conversion]
--FFMPEG Video: Efficient Splicing, Keyframes, Codecs, Transcoding
--FFMPEG Audio: Sampling Frequency, Sound Normalization

[Misc]
--Automating the Production of quality memes
--Teamwork and Coding while sleep-deprived

## What's next for Wubba Lubba Dubz

We'd like to incorporate a GUI with a slider, to more accurately adjust start/end times for each word.
Right now, we can only identify words which have been spoken exactly as entered. With Nikhil's background in linguistics, we will split an unknown word into its phonetic components.
Ideally, we will build a neural net which allows use to choose the best sound file for each word (in context).