## Inspiration

Nutritional labels tell you about the % of the daily recommended nutrients and vitamins the food you consume provides you with. Now, what if there was also a way to track the environmental impact of the food you consume?
The Footprint app allows its users to scan grocery store items and get their estimated carbon and water footprints so that they can make informed decisions regarding the food they buy. The app also allows its users to track their carbon and water footprints over time, to see the impact of their changes in lifestyle and diet. We aim to make a more sustainable life! :joy:

## What it does

There are multiple functions inside the app, but our goal is to make it as simple as possible for the users. Here are the steps:

1. Open the app to the home page
2. Press "SCAN PRODUCT" button, which will navigate to the product scanner page
3. On the scanner page, users can either use the camera to take picture of grocery products or select an image from their phone album.
4. Not satisfied with the photo? No worries! We have a retake option!
5. The "Use Photo" button on the button right navigates the user to the carbon footprint and water footprint data and the comparison with similar grocery items.
This is accomplished by prediction the product using image classification. The image is uploaded to the cloud and processed. The prediction data is stored in output file that shows the grocery item detected.
6. On the home page, the other button "VIEW PROFILE" links to a user's personal account to keep track of their carbon footprint and output a pie chart for different products that are documented. It also shows the percentage difference of the user's carbon footprint compared to that of the average American citizen.

## How we built it

First, we brainstormed about the topic: "Sustainability" for innovative ideas and solutions. Later, we benchmarked through current apps in the genre for more inspiration and understand what is insufficient in the market that we could contribute to.

A Convenient and Informative Tool to Raise the Awareness of Sustainability

To save the user's time, our goal is to make it as handy as possible. We implement image recognition, since vision is the most straightforward information, and prediction, to save customer's time to type in brands and search for the right product. Charts and plots are shown for effortless understanding of carbon footprint and water footprint data.
After having a direction to proceed, we started gathering data for data analysis. The datasets contain carbon and water footprint information.
JavaScript is used to construct front-end coding and python in addition to ImageAI are used to compute the prediction. The app is made via expo and react.

## Challenges we ran into

Some challenges we ran into involved being able to integrate our image classification model with JavaScript code and being able to run image classification real-time on a mobile device.

## Accomplishments that we're proud of

For this project, all team members stepped out of their comfort zones and pushed themselves to work in unfamiliar areas all while having fun!! We are proud to have created a product that aims to bring sustainability awareness to the general population and are proud at how far we progressed in the development of our app.

## What we learned

From a general perspective, we worked with data related to carbon and water footprints, and in the process, discovered interesting and shocking facts related to carbon consumption and emission and water usage. From a technical perspective, some members of the team had never worked with web or mobile development before, so we learned much about the coding details and all the new and exciting web development technologies that are out there. Additionally, we also delved into the existing research, models, and approaches in the field of grocery and retail image classification.

## What's next for Footprint

Although we are proud of our work, there are a few functions that we would like to improve in the future:

* Expand database and train more data for prediction
* Upload real-time image to local server for faster image detection
* Real-time video for camera and real-time image detection
* Augmented reality icons to show if grocery items are sustainable
## Inspiration

In high school, a teacher of ours used to sit in the middle of the discussion to draw lines from one person to another on paper to identify the trends of the discussion. It's a very meaningful activity as we could see how balanced the discussion is, allowing us to highlight to people who had less chance to express their ideas. It could also be used by teacher in earlier education, to identify social challenges such as anxiety, speech disorder in children.

## What it does

The app initially is trained on a short audio from each of the member of the discussion. Using transfer learning, it will able to recognize the person talking. And, during discussion, very colorful and aesthetic lines will began drawing from person to another on REAL-TIME!

## How we built it

On the front-end, we used react and JavaScript to create a responsive and aesthetic website. Vanilla css (and a little bit of math, a.k.a, Bézier curve) to create beautiful animated lines, connecting different profiles.
On the back-end, python and tensorflow was used to train the AI model. First, the audios are pre-processed into smaller 1-second chunks of audio, before turning them into a spectrogram picture. With this, we performed transfer learning with VGG16, to extract features from the spectrograms. Then, the features are used to fit a SVM model, using scikit-learn. Subsequently, the back-end opens a web-socket with the front-end to receive stream of data, and return label of the person talking. This is also done with multi-threading to ensure all the data is being processed quickly.

## Challenges we ran into

As it's our first time with deep-learning or training an AI for that matter, it was very difficult to get started. Despite the copious amount of resources and projects done, it was hard to identify a suitable source. Different processing techniques were also needed to know, before the model could be trained. In additions, finding a platform (such as Google Colab) was also necessary to train the model in appropriate time. Finally, it was fairly hard to incorporate the project with the rest of the project. It needs to be able to process the data in real-time, while keeping the latency low.

Another major challenge that we ran into was connecting the back-end with the front-end. As we wanted it to be real-time, we had to be able to stream the raw data to the back-end. But, there were problems reconstructing the binary files into appropriate format, because we were unsure what format RecordRTC uses to record audio. There was also a problem of how much data or how frequent the data should be sent over due to our high latency of predicting (~500ms). It's a problem that we couldn't figure out in time

## Accomplishments that we're proud of

The process of training the model was really cool!!! We could never think of training a voice recognition model similar to how you would to to an image/face recognition. It was a very out-of-the-box method, that we stumbled up online. It really motivated us to get out there and see what else. We were also fairly surprised to get a proof-of-concept real-time processing with local audio input from microphone. We had to utilize threading to avoid overflowing the audio input buffer. And if you get to use threading, you know it's a cool project :D.

## What we learned

Looking back, the project was quite ambitious. BUT!! That's how we learned. We learned so much about training machine learning as well as different connection protocols over the internet. Threading was also a "wet" dream of ours, so it was really fun experimenting with the concept in python.

## What's next for Hello world

The app would be much better on mobile. So, there are plans to port the entire project to mobile (maybe learning React Native?). We're also planning on retrain the voice recognition model with different methods, and improve the accuracy as well as confidence level. Lastly, we're planning on deploying the app and sending it back to our high school teacher, who was the inspiration for this project, as well as teachers around the world for their classrooms.

## Sources

These two sources helped us tremendously in building the model:

<https://medium.com/@omkarade9578/speaker-recognition-using-transfer-learning-82e4f248ef09>

<https://towardsdatascience.com/automatic-speaker-recognition-using-transfer-learning-6fab63e34e74>
## Inspiration

Our project was inspired by geocaching. Geocaching is the idea of crowdsourcing scavenger hunts where users can hide caches throughout a city and other users can go on adventures to discover them.

Team:
Dee Lucic
Matt Mitchell
Keshav Chawla
Alexei dela Pena

For this project we wanted to use the follow concepts:

1. The idea of scavenger hunts / finding things in the environment
2. Staying active and "taking care of yourself" to support Wealthsimple's motto
3. To go on an adventure, and to "push your limits" with Mountain Dew's motto
4. To use Indico image recognition API to recognize these scavenger hunt objects
5. Use Twitter to share photos of scavenger hunt objects

## What it does

We have developed a marketing tool that is a Pepsi/Mountain Dew app that gives the users a list of tasks (scavenger hunt items) that the user needs to find in the environment. The tasks satisfy an active lifestyle like taking a picture of a ping pong ball. When the user finds the item, they take a picture of it and the app will verify if the item is correct through image recognition. If enough items are found, then the app will unlock a DewTheDew fridge, which will vend free Mountain Dew to the lucky winner. All photos are posted on Twitter under the hastag: DoForDew

## How I built it

Our app is an iOS app built in Swift. It communicates with our local PHP webservice that is hosted on a laptop. The iOS app also tweets our photos. The PHP webservice calls python scripts that call the Indico image recognition API. The webservice also sends serial commands on the USB port to the fridge which contains an arduino. The arduino is programmed to process serial commands and control an analog servo motor that locks and unlocks the door of the fridge.

## Challenges I ran into

We initially tried to communicate directly with the arduino from the iOS app using bluetooth but the arduino bluetooth libraries, and the iOS bluetooth libraries are complicated, and we would not have enough time to implement the code in time.

## Accomplishments that I'm proud of

The app developer is new to iOS and Swift and we are happy with what we have created so far. The webservice is very powerful and it can handle different expansions if required. To make the fridge we had to go to a local hardware store and find the parts to create a working model in time. The challenge was difficult but we are happy with the progress so far.

## What I learned

All the developers on our team are inexperienced and we are happy that we were able to integrate all these in just a day's worth of development. We learned how to make UIs in iOS, how to pass information over PHP, Serial communications, and REST APIs.

## What's next for DoTheDew HackTheFridge

The next thing to do is to integrate bluetooth into the iOS app and set up a bluetooth shield on the arduino. Also, the Inico image recognition APIs can be called from the iOS app too. Once these two changes are made, then we will no longer require the webservice server and we can get rid of the laptop. That's the first step to creating a standalone product. After that we'll be able to create a real fridge.