## Inspiration

The inspiration for PillHero was to create a complete solution to helping the sick and elderly to learn and stick too their schedule for taking medication.

## What it does

Using the iOS app, the user enters their medication schedule, which is composed of a start date, end date, daily timings, and the medication name. The user places their medication on the pill mount. As the time nears for the user to take their medication, the app notifies the user that they should take their medication. The pill mount detects when they take their medication. If the user does not, the app notifies them. Should the user not take the pill within a specified period after this, it is counted as them having missed taking their medication.

## How we built it

-- Hardware --
The pill mount was built out of three main components: the Raspberry Pi 3, an infrared LED, and an infrared phototransistor. The LED and phototransistor are used as a touch-less way of detecting when the user removes the pill bottle from the mount. If this was done close enough to when the user is supposed to take their pills, then the assumption is made that the user took their pills for that time slot.
-- Software --
On the software side there was a combination of front end as well as back end developments. For the front end, an iOS mobile application was used to gain input from the user as well as to present calendar information. The input method utilizes speech-to-text technology to translate user voice input to a text input. From there, the user is able to edit the textfield if any mistakes were made. Then to input is passed to an AI for parsing and understanding of the statement. That data is later is sent to a back end server that keeps track of the user's pill intake via communication with the Raspberry Pi 3 and communicating using the calendar screen on the iOS application.

## Challenges we ran into

one of the major challenges we encountered was trying to connect the raspberry pi, the phone and the server all together. Due to the security of the wifi, it was very difficult to ssh into either the raspberry pi or properly set up http requests between the devices. Our work around for this was to set up a LAN that let us connect all our devices and easily make requests between them

## Accomplishments that we're proud of

Completing a project that involves hardware has been a goal of multiple team members for a while and this hackathon gave them the opportunity to do that. In reality though, the team is most proud that they were able to create an internet of things project where the hardware and software complement each other rather than feel arbitrarily attached.

## What we learned

The entire team learned a significant amount about network programming from troubleshooting issues with the Pi 3 and trying to interface the server with both system components. Otherwise, the learning was different for each member. The members who had experience with hardware were exposed to working with servers and databases. Simultaneously, the members with full-stack experience gained exposure to working with hardware. This resulted in all members becoming more well-rounded developers and gaining experience that will aid them in future hackathons and their careers.

## What's next for PillHero

Post-hackathon, the team plans to continue development of PillHero into a more complete product. Numerous upgrades are planned like the ability to track non-daily medication routines and functionality with multiple drugs. On the iOS side in particular, implementation of push notifications and text message reminders are planned. As for the pill mount, the team plans to add an accurate to team plans to add an accurate analog scale to the device so it can notify the user when they are running out of pills.
## Inspiration

Imagine you broke your EpiPen but you need it immediately for an allergic reaction. Imagine being lost in the forest with cut wounds and bleeding from a fall but have no first aid kit. How will you take care of your health without nearby hospitals or pharmacies? Well good thing for you, we have **MediFly**!! MediFly is inspired by how emergency vehicles such as ambulances take too long to get to the person in need of aid because of other cars on the road and traffic. Every second spent waiting is risking someone's life. So in order to combat that issue, we use **drones** as the first emergency responders to send medicine to save people's lives or keep them in a stable condition before human responders arrive.

## What it does

MediFly allows the user to request for emergency help or medication such as an Epipen and Epinephrine. First you download the MediFly app and create a personal account. Then you can log into your account and use the features when necessary. If you are in an emergency, press the "EMERGENCY" button and a list of common medication options will appear for the person to pick from. There is also an option to search for your needed medication. Once a choice is selected, the local hospital will see the request and send a drone to deliver the medication to the person. Human first responders will also be called. The drone will have a GPS tracker and a GPS location of the person it needs to send the medication to. When the drone is within close distance to the person, a message is sent to tell them to go outside to where the drone can see the person. The camera will use facial recognition to confirm the person is indeed the registered user who ordered the medication. This level of security is important to ensure that the medication is delivered to the correct person. When the person is confirmed, the medication holding compartment lid is opened so the person can take their medication.

## How we built it

On the software side, the front end of the app was made with react coded in Javascript, and the back end was made with Django in Python.
The text messages work through Twilio. Twilio is used to tell the user that the drone is nearby with the medication ready to hand over. It sends a message telling the person to go outdoors where the drone will be able to find the user.
On the hardware side, there are many different components that make up the drone. There are four motors, four propeller blades, a electronic speed controller, a flight controller, and 3D printed parts such as the camera mount, medication box holder, and some components of the drone frame. Besides this there is also a Raspberry Pi SBC attached to the drone for controlling the on-board systems such as the door to unload the cargo bay and stream the video to a server to process for the face recognition algorithm.

## Challenges we ran into

Building the drone from scratch was a lot harder than we anticipated. There was a lot of setting up that needed to be done for the hardware and the building aspect was not easy. It consisted of a lot of taking apart, rebuilding, soldering, cutting, hot gluing, and rebuilding.

Some of the video streaming systems did not work well at first, due to the CORS blocking the requests, given that we were using two different computers to run two different servers.

Traditional geolocation techniques often take too long - as such, we needed to build a scheme to cache a user's location before they decided to send a request to prevent lag. Additionally, the number of pages required to build, stylize, and connect together made building the site a notable challenge of scale.

## Accomplishments that we're proud of

We are extremely proud of the way the drone works and how it's able to move at quick, steady speeds while carrying the medication compartment and battery.
On the software side, we are super proud of the facial recognition code and how it's able to tell the difference between different peoples' faces. The front and back end of the website/app is also really well done. We first made the front end UI design on Figma and then implemented the design on our final website.

## What we learned

For software we learned how to use React, as well as various user authorization and authentication techniques. We also learned how to use Django.
We learnt how to build an accurate, efficient and resilient face detection recognition and tracking system to make sure the package is always delivered to the correct person.
We experimented with and learned various ways to stream real-time video over a network, also over longer ranges for the drone.
For hardware we learned how to set up and construct a drone from scratch!

## What's next for MediFly

In the future we hope to add a GPS tracker to the drone so that the person who orders the medication can see where the drone is on its path.
We would also add Twilio text messages so that when the drone is within a close radius to the user, it will send a message notifying the person to go outside and wait for the drone to deliver the medication.
## Inspiration

Hi! I am a creator of Snapill. Recently, I had to undergo wisdom tooth surgery. The surgery wasn't that bad at all, however, I was prescribed around 5 different medications, from Tylenol to NSAIDs to Steroids. It was hard to keep track of all these medications, and as a result, I ended up going through more pain during my recovery time.

However, to other people, the punishment of medication error is far more dangerous. Around half of drug-related deaths in the US is related to prescription drug use and error (22,000 people, NCBI). Furthermore, it is much harder for older people or those going through complex diseases/surgeries to keep track of their medication requirements. My personal experience, along with these pressing issues, has motivated me to design **Snapill**, a Computer-Vision based Medication App, in order to improve patient safety and awareness towards the power a single pill has.

## What it does

You may notice Snapill is very similar to the popular messaging app Snapchat! This was intentional, as our goal is to ... . Asides from the UI, Snapill is very powerful and differs from a common medication app.

Specifically, Snapill is the first app that leverages advanced Computer Vision techniques like homography with your common LLM. The combination allows for robustness and covers several user failure points (mainly: bad lighting). Snapill allows a patient to scan their medication vial, and automatically generates important metadata like prescription name, expiration date, dosage, and possible times to take the pill. Furthermore, we believe awareness is a strong factor for increased patient safety, so we incorporated the Cerebras AI Inference models to allow users to "chat" with their medications (we weren't joking when we alluded to SnapChat). Users are able to request more information about the drug through the chat, along with a personal vanguard that scans for incompatibility issues between medications. Finally, Snapill provides reminders when it is time for users to take their medications (we couldn't forget that part, could we?)

## How we built it

Knowing we needed data, me and Shishir worked with Scriptpro, a pharmaceutical company in Kansas. After pitching our ideas to a senior software engineer (a warm reception), we were able to acquire around 20 vials with different medication names. They were used to train both a Roboflow model to detect labels, and test a custom OpenCV unwarping pipeline.

Snapill is a combination of a react-native app ran on the user's phone that connects to a Flask backend hosted on Heroku. This allows us to use homemade openCV functions while calling APIs such as Roboflow, Cerebras, and Firebase. User data, auth, and storage is all stored through firebase, allowing for simple public downloads from other APIs. We tested the app using both an Android simulator and a physical iPhone. See below for more information about our backend process!

## Challenges we ran into

Our first challenge was that UPenn did not allow direct connection in a way. Specifically, we weren't able to run a local backend on the computer and forward its port so a phone could send requests on the same WiFi. This led us to deploy the backend on Heroku, which unfortunately slowed down build times but taught us a lot about deploying on production lines!

Another challenge we ran into was integrating our custom OpenCV code with Roboflow's Workflow design. However, it made us glad we chose to use a Python backend to process Computer Vision! We solved this issue by sandwiching our custom unwarp code between two different Roboflow Workflows.

## Accomplishments that we're proud of

The main accomplishment we're proud of was our CV pipeline. Regularly, OCR (Optical Character Recognition) is used for parsing data from images, however we noticed it has a very high error rate when trying to detect text on curved surfaces. To solve this, me and Shishir researched and utilized a pipeline that uses concepts from linear algebra to essentially map the text on the curved label to a flat rectangle. This wasn't easy because there were very few resources for using OCR in edge cases like this. But we curved around (pun not intended) and explored the depths of OpenCV to achieve our goal. OCR was much more accurate now and was able to safely parse label data.

Even cooler, is that it could be scaled to pretty much any text on the curved part of a cylinder! All the method needs are certain extreme points, and the rest is up to interpolation. After Pennapps, I plan to test the pipeline more, and see how to automatically detect the points using edge/contour recognition.

## What we learned

For one of us, this was his first time both attending a hackathon AND actually creating a mobile app. While he knew React from previous projects, he still had to learn how to use expo along with several other libraries, and was able to pull it off! We all also learnt about how important reliability is when making patient technology solutions. Handling important data like dosage should be done so carefully.

## What's next for Snapill

The next update for Snapill would be finding a way to make the refilling process easier. Usually, there are phone numbers on vial labels, which would be a good next step to detect.