## Inspiration

Currently, there is an exponential growth of obesity in the world, leading to devastating consequences such as an increased rate of diabetes and heart diseases. All three of our team members are extremely passionate about nutrition issues and wish to educate others and promote healthy active living.

## What it does

This iOS app allows users to take pictures of meals that they eat and understand their daily nutrition intake. For each food that is imaged, the amount of calories, carbohydrates, fats and proteins are shown, contributing to the daily percentage on the nutrition tab. In the exercise tab, the users are able to see how much physical activities they need to do to burn off their calories, accounting for their age and weight differences. The data that is collected easily syncs with the iPhone built-in health app.

## How we built it

We built the iOS app in Swift programming language in Xcode. For the computer vision of the machine learning component, we used CoreML, and more specifically its Resnet 50 Model. We also implemented API calls to Edamam to receive nutrition details on each food item.

## Challenges we ran into

Two of our three team members have never used Swift before - it is definitely a challenge writing in an unfamiliar coding language. It was also challenging calling different APIs and integrating them back in Xcode, as the CoreML documentation is unclear.

## Accomplishments that we're proud of

We are proud of learning an entirely new programming language and building a substantial amount of a well-functioning app within 36 hours.

## What's next for NutriFitness

Building our own machine learning model and getting more accurate image descriptions.
## Inspiration

We were inspired by the health vertical and Vision Recognition technologies. We wanted to create a very simple and user friendly way to track calorie and nutrient intakes, hoping to make it as easy as possible to develop and sustain healthy eating habits.

## About Our Product

Using Clarifai's food model vision recognition, NutriBot's mobile app detects the types of food present in a picture the users take on their phones. It then prompts the user with a list of predictions of the contents of the image and asks the user to choose the correct ones among the list. Then, using Nutritionix API, it gives a breakdown of the calories and nutrition values for each food item. On a separate web app, the historical data is displayed to the users to inform them about their long term trends in food consumption.

## How We Built It

The mobile app was built using React Native, Clarifai Food Model API, Nutritionix API, and Expo.
The web app was built using Wix Code.

## Challenges

Finding the right platform took multiple trial-and-error. We explored everything from Flask to Wix Code and graphing applications like Charts Ninja and chart.bundle.js. As most of us were unfamiliar with these new platforms and JavaScript, our biggest challenge was learning those skills and being able to apply them in a timely, scalable manner.

## Accomplishments

Although our app still has many more features to come, it well encompasses our initial vision with our 36 hours of hacking. We took every hour as an opportunity to learn by struggling, exploring, and collaborating.

## What We Learned

Every expert starts out as a beginner! Coming in, we didn't know everything we needed to know for our project. But through persistence and proactiveness, we were able to turn the gaps in our knowledge into a motivating source for learning. From this weekend, we are taking away Wix Code, JavaScript, Flask, HTML, and React Native, which we hope to then apply to our future hacking endeavors.

## What's Next for NutriBot

We aim to be able to give users a more thorough analysis of their food consumption trends such as their daily calorie intakes, calorie values broken down by specific nutrition values like protein, carbohydrate, fats on weekly and monthly bases. We also hope to be able to give users instant feedback about their meal plates, for example if the user's plate only has fat-heavy food, we want to display a warning that suggests them to consume more vegetables and/or protein.
## Inspiration

While there are several applications that use OCR to read receipts, few take the leap towards informing consumers on their purchase decisions. We decided to capitalize on this gap: we currently provide information to customers about the healthiness of the food they purchase at grocery stores by analyzing receipts. In order to encourage healthy eating, we are also donating a portion of the total value of healthy food to a food-related non-profit charity in the United States or abroad.

## What it does

Our application uses Optical Character Recognition (OCR) to capture items and their respective prices on scanned receipts. We then parse through these words and numbers using an advanced Natural Language Processing (NLP) algorithm to match grocery items with its nutritional values from a database. By analyzing the amount of calories, fats, saturates, sugars, and sodium in each of these grocery items, we determine if the food is relatively healthy or unhealthy. Then, we calculate the amount of money spent on healthy and unhealthy foods, and donate a portion of the total healthy values to a food-related charity. In the future, we plan to run analytics on receipts from other industries, including retail, clothing, wellness, and education to provide additional information on personal spending habits.

## How We Built It

We use AWS Textract and Instabase API for OCR to analyze the words and prices in receipts. After parsing out the purchases and prices in Python, we used Levenshtein distance optimization for text classification to associate grocery purchases with nutritional information from an online database. Our algorithm utilizes Pandas to sort nutritional facts of food and determine if grocery items are healthy or unhealthy by calculating a “healthiness” factor based on calories, fats, saturates, sugars, and sodium. Ultimately, we output the amount of money spent in a given month on healthy and unhealthy food.

## Challenges We Ran Into

Our product relies heavily on utilizing the capabilities of OCR APIs such as Instabase and AWS Textract to parse the receipts that we use as our dataset. While both of these APIs have been developed on finely-tuned algorithms, the accuracy of parsing from OCR was lower than desired due to abbreviations for items on receipts, brand names, and low resolution images. As a result, we were forced to dedicate a significant amount of time to augment abbreviations of words, and then match them to a large nutritional dataset.

## Accomplishments That We're Proud Of

Project Horus has the capability to utilize powerful APIs from both Instabase or AWS to solve the complex OCR problem of receipt parsing. By diversifying our software, we were able to glean useful information and higher accuracy from both services to further strengthen the project itself, which leaves us with a unique dual capability.

We are exceptionally satisfied with our solution’s food health classification. While our algorithm does not always identify the exact same food item on the receipt due to truncation and OCR inaccuracy, it still matches items to substitutes with similar nutritional information.

## What We Learned

Through this project, the team gained experience with developing on APIS from Amazon Web Services. We found Amazon Textract extremely powerful and integral to our work of reading receipts. We were also exposed to the power of natural language processing, and its applications in bringing ML solutions to everyday life. Finally, we learned about combining multiple algorithms in a sequential order to solve complex problems. This placed an emphasis on modularity, communication, and documentation.

## The Future Of Project Horus

We plan on using our application and algorithm to provide analytics on receipts from outside of the grocery industry, including the clothing, technology, wellness, education industries to improve spending decisions among the average consumers. Additionally, this technology can be applied to manage the finances of startups and analyze the spending of small businesses in their early stages. Finally, we can improve the individual components of our model to increase accuracy, particularly text classification.