## Inspiration

Hi! I am a creator of Snapill. Recently, I had to undergo wisdom tooth surgery. The surgery wasn't that bad at all, however, I was prescribed around 5 different medications, from Tylenol to NSAIDs to Steroids. It was hard to keep track of all these medications, and as a result, I ended up going through more pain during my recovery time.

However, to other people, the punishment of medication error is far more dangerous. Around half of drug-related deaths in the US is related to prescription drug use and error (22,000 people, NCBI). Furthermore, it is much harder for older people or those going through complex diseases/surgeries to keep track of their medication requirements. My personal experience, along with these pressing issues, has motivated me to design **Snapill**, a Computer-Vision based Medication App, in order to improve patient safety and awareness towards the power a single pill has.

## What it does

You may notice Snapill is very similar to the popular messaging app Snapchat! This was intentional, as our goal is to ... . Asides from the UI, Snapill is very powerful and differs from a common medication app.

Specifically, Snapill is the first app that leverages advanced Computer Vision techniques like homography with your common LLM. The combination allows for robustness and covers several user failure points (mainly: bad lighting). Snapill allows a patient to scan their medication vial, and automatically generates important metadata like prescription name, expiration date, dosage, and possible times to take the pill. Furthermore, we believe awareness is a strong factor for increased patient safety, so we incorporated the Cerebras AI Inference models to allow users to "chat" with their medications (we weren't joking when we alluded to SnapChat). Users are able to request more information about the drug through the chat, along with a personal vanguard that scans for incompatibility issues between medications. Finally, Snapill provides reminders when it is time for users to take their medications (we couldn't forget that part, could we?)

## How we built it

Knowing we needed data, me and Shishir worked with Scriptpro, a pharmaceutical company in Kansas. After pitching our ideas to a senior software engineer (a warm reception), we were able to acquire around 20 vials with different medication names. They were used to train both a Roboflow model to detect labels, and test a custom OpenCV unwarping pipeline.

Snapill is a combination of a react-native app ran on the user's phone that connects to a Flask backend hosted on Heroku. This allows us to use homemade openCV functions while calling APIs such as Roboflow, Cerebras, and Firebase. User data, auth, and storage is all stored through firebase, allowing for simple public downloads from other APIs. We tested the app using both an Android simulator and a physical iPhone. See below for more information about our backend process!

## Challenges we ran into

Our first challenge was that UPenn did not allow direct connection in a way. Specifically, we weren't able to run a local backend on the computer and forward its port so a phone could send requests on the same WiFi. This led us to deploy the backend on Heroku, which unfortunately slowed down build times but taught us a lot about deploying on production lines!

Another challenge we ran into was integrating our custom OpenCV code with Roboflow's Workflow design. However, it made us glad we chose to use a Python backend to process Computer Vision! We solved this issue by sandwiching our custom unwarp code between two different Roboflow Workflows.

## Accomplishments that we're proud of

The main accomplishment we're proud of was our CV pipeline. Regularly, OCR (Optical Character Recognition) is used for parsing data from images, however we noticed it has a very high error rate when trying to detect text on curved surfaces. To solve this, me and Shishir researched and utilized a pipeline that uses concepts from linear algebra to essentially map the text on the curved label to a flat rectangle. This wasn't easy because there were very few resources for using OCR in edge cases like this. But we curved around (pun not intended) and explored the depths of OpenCV to achieve our goal. OCR was much more accurate now and was able to safely parse label data.

Even cooler, is that it could be scaled to pretty much any text on the curved part of a cylinder! All the method needs are certain extreme points, and the rest is up to interpolation. After Pennapps, I plan to test the pipeline more, and see how to automatically detect the points using edge/contour recognition.

## What we learned

For one of us, this was his first time both attending a hackathon AND actually creating a mobile app. While he knew React from previous projects, he still had to learn how to use expo along with several other libraries, and was able to pull it off! We all also learnt about how important reliability is when making patient technology solutions. Handling important data like dosage should be done so carefully.

## What's next for Snapill

The next update for Snapill would be finding a way to make the refilling process easier. Usually, there are phone numbers on vial labels, which would be a good next step to detect.
## Inspiration

This project was inspired by a team member’s family, his grandparents always have to take medicine but often forget about it. Not only his grandparents forget the medicine also his mom. Although, his mom is very young but in a very fast paced society nowadays people always forget to do small things like taking their pills. Due to this inspiration, we decided to develop a pill reminder, but then we got inspired by a Tik Tok video about a person who has Parkinson’s disease and he couldn’t pick up an individual pill from the container. In end, we decide to create this project that will resolve the problem of people forgetting to take their pills as well as helping people to easily take individual pills.

## What it does

Our project the Delta Dispenser uses an app to communicate with the database to set up a specific time to alert users to take their pills as well as tracking their pills information in the app. The hardware of Delta Dispenser will alert the user when the time is reached and automatically dispense the correct amount of the pills into the container.

## How we built it

The frontend of the app is made with **Flutter**, the app communicates with a **firebase real-time database** to store medicinal and scheduling information for the user. The physical component uses an **embedded microcontroller** called an ESP-32 which we chose for its ability to connect to WiFi and be able to sync with the firebase database to know when to dispense the pills.

## Challenges we ran into

The time constraint was definitely a big challenge and we accounted for that by deciding which features were most important in emphasizing our main idea for this project. These parts include the mechanical indexer of the pills, the interface the user would interact with, and how the database would look for communication with the app and the embedded device.

## Accomplishments that we're proud of

We are most proud of how this project utilized many different aspects of engineering, from mechanical to electrical and software. Our team did a really good job at communicating throughout the design process which made integration at the end much easier.

## What we learned

During this project, we had learned how to flutter to create a mobile app as well as learning how firebase works. Throughout this project, although we only learned a few skills that will be very useful in the future. The most important part was that we were able to develop upon the skills we already had. For example, now we are able to develop hardware that could communicate through firebase.

## What's next for Delta Dispenser

The next steps for the Delta Dispenser include building a fully 3D printed prototype, along with the control box and hopper as shown in the CAD renders. On the software side, we would also like to add the ability for more complicated drug scheduling, while keeping the UI easy enough for anyone to set up. Having another portal that allows a doctor to directly input the information themselves is also a feature we are interested in having.
View presentation at the following link: <https://youtu.be/Iw4qVYG9r40>

## Inspiration

During our brainstorming stage, we found that, interestingly, two-thirds (a majority, if I could say so myself) of our group took medication for health-related reasons, and as a result, had certain external medications that result in negative drug interactions. More often than not, one of us is unable to have certain other medications (e.g. Advil, Tylenol) and even certain foods.
Looking at a statistically wider scale, the use of prescription drugs is at an all-time high in the UK, with almost half of the adults on at least one drug and a quarter on at least three. In Canada, over half of Canadian adults aged 18 to 79 have used at least one prescription medication in the past month. The more the population relies on prescription drugs, the more interactions can pop up between over-the-counter medications and prescription medications. Enter Medisafe, a quick and portable tool to ensure safe interactions with any and all medication you take.

## What it does

Our mobile application scans barcodes of medication and outputs to the user what the medication is, and any negative interactions that follow it to ensure that users don't experience negative side effects of drug mixing.

## How we built it

Before we could return any details about drugs and interactions, we first needed to build a database that our API could access. This was done through java and stored in a CSV file for the API to access when requests were made. This API was then integrated with a python backend and flutter frontend to create our final product. When the user takes a picture, the image is sent to the API through a POST request, which then scans the barcode and sends the drug information back to the flutter mobile application.

## Challenges we ran into

The consistent challenge that we seemed to run into was the integration between our parts.
Another challenge that we ran into was one group member's laptop just imploded (and stopped working) halfway through the competition, Windows recovery did not pull through and the member had to grab a backup laptop and set up the entire thing for smooth coding.

## Accomplishments that we're proud of

During this hackathon, we felt that we *really* stepped out of our comfort zone, with the time crunch of only 24 hours no less. Approaching new things like flutter, android mobile app development, and rest API's was daunting, but we managed to preserver and create a project in the end.
Another accomplishment that we're proud of is using git fully throughout our hackathon experience. Although we ran into issues with merges and vanishing files, all problems were resolved in the end with efficient communication and problem-solving initiative.

## What we learned

Throughout the project, we gained valuable experience working with various skills such as Flask integration, Flutter, Kotlin, RESTful APIs, Dart, and Java web scraping. All these skills were something we've only seen or heard elsewhere, but learning and subsequently applying it was a new experience altogether. Additionally, throughout the project, we encountered various challenges, and each one taught us a new outlook on software development. Overall, it was a great learning experience for us and we are grateful for the opportunity to work with such a diverse set of technologies.

## What's next for Medisafe

Medisafe has all 3-dimensions to expand on, being the baby app that it is. Our main focus would be to integrate the features into the normal camera application or Google Lens. We realize that a standalone app for a seemingly minuscule function is disadvantageous, so having it as part of a bigger application would boost its usage. Additionally, we'd also like to have the possibility to take an image from the gallery instead of fresh from the camera. Lastly, we hope to be able to implement settings like a default drug to compare to, dosage dependency, etc.