# Healthy.ly

An android app that can show if a food is allergic to you just by clicking its picture. It can likewise demonstrate it's health benefits, ingredients, and recipes.

## Inspiration

We are a group of students from India. The food provided here is completely new to us and we don't know the ingredients. One of our teammates is dangerously allergic to seafood and he has to take extra precautions while eating at new places. So we wanted to make an app that can detect if the given food is allergic or not using computer vision.

We also got inspiration from the HBO show **Silicon Valley**, where a guy tries to make a **Shazam for Food** app.

Over time our idea grew bigger and we added nutritional value and recipes to it.

## What it does

This is an android app that uses computer vision to identify food items in the picture and shows you if you are allergic to it by comparing the ingredients to your restrictions provided earlier. It can also give the nutritional values and recipes for that food item.

## How we built it

We developed a deep learning model using **Tensorflow** that can classify between 101 different food items. We trained it using the **Google Compute Engine** with 2vCPUs, 7.5 GB RAM and 2 Tesla K80 GPU. This model can classify 101 food items with over 70% accuracy.


From the predicted food item, we were able to get its ingredients and recipes from an API from rapidAPI called "Recipe Puppy". We cross validate the ingredients with the items that the user is allergic to and tell them if it's safe to consume.


We made a native **Android Application** that lets you take an image and uploads it to **Google Storage**. The python backend runs on **Google App Engine**. The web app takes the image from google storage and using **Tensorflow Serving** finds the class of the given image(food name). It uses its name to get its ingredients, nutritional values, and recipes and return these values to the android app via **Firebase**.


The Android app then takes these values and displays them to the user. Since most of the heavy lifting happens in the cloud, our app is very light(7MB) and is **computationally efficient**. It does not need a lot of resources to run. It can even run in a cheap and underperforming android mobile without crashing.

## Challenges we ran into

>
> 1. We had trouble converting our Tensorflow model to tflite(tflite\_converter could not convert a multi\_gpu\_model to tflite). So we ended up hosting it on the cloud which made the app lighter and computationally efficient.
> 2. We are all new to using google cloud. So it took us a long time to even figure out the basic stuff. Thanks to the GCP team, we were able to get our app up and running.
> 3. We couldn't use the Google App Engine to support TensorFlow(we could not get it working). So we have hosted our web app on Google Compute Engine
> 4. We did not get a UI/UX designer or a frontend developer in our team. So we had to learn basic frontend and design our app.
> 5. We could only get around 70% validation accuracy due to the higher computation needs and less available time.
> 6. We were using an API from rapidAPI. But since yesterday, they stopped support for that API and it wasn't working. So we had to make our own database to run our app.
> 7. Couldn't use AutoML for vision classification, because our dataset was too large to be uploaded.
>
>
>

## What we learned

Before coming to this hack, we had no idea about using cloud infrastructure like Google Cloud Platform. In this hack, we learned a lot about using Google Cloud Platform and understand its benefits. We are pretty comfortable using it now.

Since we didn't have a frontend developer we had to learn that to make our app.

Making this project gave us a lot of exposure to **Deep Learning**, **Computer Vision**, **Android App development** and **Google Cloud Platform**.

## What's next for Healthy.ly

1. We are planning to integrate **Google Fit** API with this so that we can get a comparison between the number of calories consumed and the number of calories burnt to give better insight to the user. We couldn't do it now due to time constraints.
2. We are planning to integrate **Augmented Reality** with this app to make it predict in real-time and look better.
3. We have to improve the **User Interface** and **User Experience** of the app.
4. Spend more time training the model and **increase the accuracy**.
5. Increase the **number of labels** of the food items.
## Inspiration

As university students, we often find that we have groceries in the fridge but we end up eating out and the groceries end up going bad.

## What It Does

After you buy groceries from supermarkets, you can use our app to take a picture of your receipt. Our app will parse through the items in the receipts and add the items into the database representing your fridge. Using the items you have in your fridge, our app will be able to recommend recipes for dishes for you to make.

## How We Built It

On the back-end, we have a Flask server that receives the image from the front-end through ngrok and then sends the image of the receipt to Google Cloud Vision to get the text extracted. We then post-process the data we receive to filter out any unwanted noise in the data.

On the front-end, our app is built using react-native, using axios to query from the recipe API, and then stores data into Firebase.

## Challenges We Ran Into

Some of the challenges we ran into included deploying our Flask to Google App Engine, and styling in react. We found that it was not possible to write into Google App Engine storage, instead we had to write into Firestore and have that interact with Google App Engine.

On the frontend, we had trouble designing the UI to be responsive across platforms, especially since we were relatively inexperienced with React Native development. We also had trouble finding a recipe API that suited our needs and had sufficient documentation.
## Inspiration

In today's fast-paced world, effective communication is more crucial than ever. Whether it's acing a job interview, delivering a compelling speech, or simply making a persuasive argument, the way we present ourselves can make all the difference. However, the lack of real-time, comprehensive feedback often hinders individuals from realizing their full potential. This inspired us to create Enterview, a tool designed to provide immediate, actionable insights into your emotional and verbal delivery.

---

## What it does

Enterview is an app that allows users to upload or record a video of themselves speaking. Using Hume AI developer tool, the app analyzes the user's facial expressions and voice to identify various emotions. It then processes this data and serves it as a prompt to OpenAI, which generates personalized feedback. The app also provides a detailed breakdown of the user's emotional shifts, tonal shifts, dominant emotions at various points, and other key metrics. This comprehensive feedback aims to help users identify areas for improvement, whether they are preparing for job interviews, public speaking, or any other situation requiring effective communication.

---

## How we built it

We used Next.js for full-stack development and used Flask, additionally, for back-end. We then integrated it with Hume's emotion and text recognition API and OpenAI's GPT-3. The architecture is designed to first record and send the video data to Hume, receive the emotion and text metrics on the server side, and then forward this processed data to OpenAI for generating insightful feedback.

---

## Challenges we ran into

1. **API Integration**: Ensuring seamless interaction between Hume and OpenAI was initially challenging due to data transfer and formatting issues.
2. **Accuracy**: We found that emotion recognition could include large amounts of data that may not be useful to the user, which led us to explore ways to define certain interpretations of the data and try and manipulate it in a way that would give us the best possible feedback.

---

## Accomplishments that we're proud of

1. **Holistic Feedback**: Successfully integrating Hume and OpenAI to provide a 360-degree view of a user's performance.
2. **User Experience**: Creating an intuitive, user-friendly interface that even those who are not tech-savvy can navigate easily.

---

## What we learned

1. **Interdisciplinary Integration**: The importance of combining different technologies to create a more comprehensive solution.
2. **Data Processing**: How to determine the usefulness of certain types of data and interpret them in a more efficient way.
3. **Ethical Considerations**: TODO

---

## What's next for Enterview

1. **Real-Time Analysis**: We plan to introduce a real-time feedback feature that provides insights as the user is speaking.
2. **Feedback customization Features**: Adding options to check for specific types of emotions, and tones and make the feedback more customizable.
3. **Expanded Use Cases**: We aim to extend the app's utility to other domains like therapy and education.