## Inspiration

Indecision is a problem a lot of people face at meal times, and we wanted to create a way to help people *decide* what meals to make at home while also being able to make use of the food they **already** have laying around. This helps to minimize food waste, while also saving you a trip to the grocery store!

## What it does

How does RecipeVision do this? Recipe Vision is a easy to use, but very powerful mobile application. You take a picture of your fridge, pantry, cabinets, etc, and our app scans it, tells you what it thinks you have, and then gives you recipes using what you have. You have the choice of taking a new picture, or using an existing picture. Take a picture, press a button, get recipes. **It's that easy.**

## How we built it

We implemented Microsoft's Cognitive API, specifically Computer Vision, to analyze the pictures taken and give us information that we could then parse and use to find recipes. With the food information given from Microsoft's algorithms, we can then search our database for recipes that can use the ingredients you have.
<https://www.microsoft.com/cognitive-services/en-us/computer-vision-api>

## Challenges we ran into

We were challenged by time constraints, so we were unable to implement a database of recipes to pull from based on the food our app detected. However, we did come up with two plans of action to tackle this. One would be to use Spoonacular's Food API, which can parse through a their huge database of recipes based on inputted ingredients.
<https://market.mashape.com/spoonacular/recipe-food-nutrition#find-by-ingredients>

The other option would be to create our own database using SQL on Azure or AWS and have more control over input parameters.
Additionally, Microsoft's Computer Vision analysis, while covering an expansive range of categories, is not always the best fit for determining the type of food in a picture. It struggles to identifies multiple foods in the same picture, and is not always completely accurate. However, we were usually able to get a good result for a picture of a single food item.

## Accomplishments that we're proud of

We are very proud that we were able to **successfully feed our smartphone images to Microsoft's Cognitive services and display the resulting analysis to the user.** This service has a lot of powerful potential uses, and we believe our app is very useful and practical.

## What we learned

We learned a ton about Microsoft's Cognitive Services and how to implement them using Java. From this experience, we got a lot of experience working with APIs made by others and reading their code thoroughly. We all expanded our knowledge of Android Studio, and worked on UI as well. It was also amazing to see how many different APIs there were out there available for public use.

## What's next for RecipeVision

We've got a lot in store. In the future, it would be great if we could implement our application onto a stationary Kinect based imaging camera mounted in the fridge and pantry, which would enable real time analysis of food stock. Another cool idea would be integration onto HoloLens, where users would be able to simply look inside their pantry and be given recipes in augmented reality based on what they are seeing.
## Inspiration

Many of us, including our peers, struggle on deciding what to cook. We usually have a fridge full of items but are not sure what exactly to make with those items. This leads us to eating outside or buying even more groceries to follow along to a recipe.

* We want to be able to use what we have
* Reduce our waste
* Get new and easy ideas

## What it does

The user first takes a picture of the items in their fridge. They can then upload the image to our application. Using computer vision technology, we detect what are the exact items present in the picture (their fridge). After obtaining a list of the ingredients the user in their fridge this data is then passed along and processed with a database of 1000 quick and easy recipes.

## How we built it

* We designed the mobile and desktop website using Figma
* The website was developed using JavaScript and node.js
* We use Google Cloud Vision API to detect items in the picture
* This list of items in then processed along a database of recipes
* Best matching recipes are returned to the user

## Challenges we ran into

We ran through a lot of difficulties and challenges while building this web app most of which we were able to overcome with help from each other and learning on the fly.

The first challenge we ran into was building and training a machine learning model to apply multi-class object detection on the images the user inputs. This is tricky as there is no proper dataset of images of vegetables, fruits, meats, condiments, other items all together. After various experiments on our own machine learning models from scratch we then attempted using multiple pre-existing models and tools for our case. We found Google Cloud Vision API was doing the best job out of all that was available. Thus, we invested in Google Vision and using their API for our prototype currently.

The second challenge was getting the correct recipes according to the data received from the artificial intelligence. We are using a database of 1000 recipes and set a threshold for the minimum of number of items needed to match (ingredients the user has - to - ingredients the recipe requires). Our assumption is the user already has the basic ingredients such as salt, pepper, salt, butter, oil, etc.

## Accomplishments that we're proud of

* Coming up with an idea that solves a problem every member of our team and many peers we interviewed face
* Using modern artificial intelligence to solve a major part of our problem (detecting ingredients/groceries) from a given image
* Designing a a very good looking and user-friendly UI with an excellent user-experience (quick and easy)

## What we learned

Each team member learned a new or enhanced a current skill during this hackathon which is what we were here for. We learned to use newer tools, such as google cloud, figma, others to streamline our product development.

## What's next for Xcellent Recipes

\**We truly believe in our product and its usefulness for customers. We will continue working on Xcellent Recipes with a product launch in the future. The next steps include: \**

1. Establishing a backend server
2. Create or obtain our own data for training a ML model for our use case
3. Fine tune recipes
4. Company Launch
## Inspiration

Unhealthy diet is the leading cause of death in the U.S., contributing to approximately 678,000 deaths each year, due to nutrition and obesity-related diseases, such as heart disease, cancer, and type 2 diabetes. Let that sink in; the leading cause of death in the U.S. could be completely nullified if only more people cared to monitor their daily nutrition and made better decisions as a result. But **who** has the time to meticulously track every thing they eat down to the individual almond, figure out how much sugar, dietary fiber, and cholesterol is really in their meals, and of course, keep track of their macros! In addition, how would somebody with accessibility problems, say blindness for example, even go about using an existing app to track their intake? Wouldn't it be amazing to be able to get the full nutritional breakdown of a meal consisting of a cup of grapes, 12 almonds, 5 peanuts, 46 grams of white rice, 250 mL of milk, a glass of red wine, and a big mac, all in a matter of **seconds**, and furthermore, if that really is your lunch for the day, be able to log it and view rich visualizations of what you're eating compared to your custom nutrition goals?? We set out to find the answer by developing macroS.

## What it does

macroS integrates seamlessly with the Google Assistant on your smartphone and let's you query for a full nutritional breakdown of any combination of foods that you can think of. Making a query is **so easy**, you can literally do it while *closing your eyes*. Users can also make a macroS account to log the meals they're eating everyday conveniently and without hassle with the powerful built-in natural language processing model. They can view their account on a browser to set nutrition goals and view rich visualizations of their nutrition habits to help them outline the steps they need to take to improve.

## How we built it

DialogFlow and the Google Action Console were used to build a realistic voice assistant that responds to user queries for nutritional data and food logging. We trained a natural language processing model to identify the difference between a call to log a food eaten entry and simply a request for a nutritional breakdown. We deployed our functions written in node.js to the Firebase Cloud, from where they process user input to the Google Assistant when the test app is started. When a request for nutritional information is made, the cloud function makes an external API call to nutrionix that provides nlp for querying from a database of over 900k grocery and restaurant foods. A mongo database is to be used to store user accounts and pass data from the cloud function API calls to the frontend of the web application, developed using HTML/CSS/Javascript.

## Challenges we ran into

Learning how to use the different APIs and the Google Action Console to create intents, contexts, and fulfillment was challenging on it's own, but the challenges amplified when we introduced the ambitious goal of training the voice agent to differentiate between a request to log a meal and a simple request for nutritional information. In addition, actually finding the data we needed to make the queries to nutrionix were often nested deep within various JSON objects that were being thrown all over the place between the voice assistant and cloud functions. The team was finally able to find what they were looking for after spending a lot of time in the firebase logs.In addition, the entire team lacked any experience using Natural Language Processing and voice enabled technologies, and 3 out of the 4 members had never even used an API before, so there was certainly a steep learning curve in getting comfortable with it all.

## Accomplishments that we're proud of

We are proud to tackle such a prominent issue with a very practical and convenient solution that really nobody would have any excuse not to use; by making something so important, self-monitoring of your health and nutrition, much more convenient and even more accessible, we're confident that we can help large amounts of people finally start making sense of what they're consuming on a daily basis. We're literally able to get full nutritional breakdowns of combinations of foods in a matter of **seconds**, that would otherwise take upwards of 30 minutes of tedious google searching and calculating. In addition, we're confident that this has never been done before to this extent with voice enabled technology. Finally, we're incredibly proud of ourselves for learning so much and for actually delivering on a product in the short amount of time that we had with the levels of experience we came into this hackathon with.

## What we learned

We made and deployed the cloud functions that integrated with our Google Action Console and trained the nlp model to differentiate between a food log and nutritional data request. In addition, we learned how to use DialogFlow to develop really nice conversations and gained a much greater appreciation to the power of voice enabled technologies. Team members who were interested in honing their front end skills also got the opportunity to do that by working on the actual web application. This was also most team members first hackathon ever, and nobody had ever used any of the APIs or tools that we used in this project but we were able to figure out how everything works by staying focused and dedicated to our work, which makes us really proud. We're all coming out of this hackathon with a lot more confidence in our own abilities.

## What's next for macroS

We want to finish building out the user database and integrating the voice application with the actual frontend. The technology is really scalable and once a database is complete, it can be made so valuable to really anybody who would like to monitor their health and nutrition more closely. Being able to, as a user, identify my own age, gender, weight, height, and possible dietary diseases could help us as macroS give users suggestions on what their goals should be, and in addition, we could build custom queries for certain profiles of individuals; for example, if a diabetic person asks macroS if they can eat a chocolate bar for lunch, macroS would tell them no because they should be monitoring their sugar levels more closely. There's really no end to where we can go with this!