## EyeSnap: Diabetic Retinopathy Detection with DiaScan - HackHarvardFall2023

### **Introduction**

Welcome to the future of healthcare, where your smartphone becomes your guardian for early diabetes detection. Our innovative solution harnesses the power of artificial intelligence to identify Diabetic Retinopathy right from the convenience of your smartphone. We are on a mission to make early diabetes diagnosis accessible and effortless for everyone.

### **Diabetic Retinopathy**

About Diabetic Retinopathy
Diabetic retinopathy is a serious eye condition that affects individuals with diabetes. It can lead to vision problems and even blindness if left untreated. This single-page website aims to provide a clear and concise explanation of diabetic retinopathy.

### **Key Components**

#### **Retinal AI**
 Retinal AI is the CNN Trained Model that can detect the presence and severity of Diabetic Retinopathy. It forms the heart of our solution, providing accurate and reliable results.

![](https://github.com/kartikey-onlineGOD/EyeSnap-HackHarvard/blob/main/Assets/image%203.png)

#### **DiaScan**
 DiaScan is a revolutionary 3D printed device that helps us scan the retina using a phone camera. The DiaScan fits around the phone like a case and allows us to take retinal scan photographs, enabling easy and non-invasive detection of diabetic retinopathy.

![](https://github.com/kartikey-onlineGOD/EyeSnap-HackHarvard/blob/main/Assets/image%202.png)

### **About Us**

Author : Kartikey Pandey

As driven Computer Science student located in Pennsylvania with a strong interest in AI and Full Stack development, I've made notable contributions within the Intel AI4Youth program.As both an apprentice and intern, I've been selected for my dedication, and I've played a crucial role in creating impactful solutions to tackle issues that a global scale. With this new solution I have focused in increase Health Equity and detect diabetes using Retinal Scans in patients.

I believe in the power of early detection and its potential to transform healthcare. Join me in this journey towards a world where everyone can have easy access to early diabetes diagnosis and treatment.

### **Story behind this Project**

I got the inspiration of this project from CareYaya was able to deeply connect with the issue as my own grandfather passed away three years ago due to Kidney Failure which was caused due to Diabetes Type 2. Listening to Neal K. Shah I was deeply moved and wanted to build a solution to this problem.

### **Contact Us**

Email: [Kartikeypandey.official@gmail.com](mailto:Kartikeypandey.official@gmail.com)

You can connect with me on Linkedin or mail me to further understand SnapEye.

Together, we can make a difference in the fight against diabetic retinopathy. Your smartphone is now your guardian in the battle for early detection and prevention.
## Inspiration

The inspiration for our application came from our own laziness and unwillingness to go to the doctors for anything. We wanted to create an application with machine learning and artificial intelligence models in order to improve the daily lives of the lazy. Our goal is to create an application to change the future of skin care and self analysis of skin conditions at home.

## What it does

Our application takes in a picture of the user's skin taken using their phone camera and uses that picture to conduct a smart inference on what skin condition the user might have based on our trained machine learning model that uses image classification.

## How we built it

For our front end, we used android and java to build the ultimate experience for the mobile user. We also used android's built in camera in order to snap a picture of their skin. For the machine learning model, we used Microsoft Azure cloud computing platform. More specifically, the custom vision AI.

## Challenges we ran into

One challenge we ran into was figuring out how to use TensorFlow to integrate the trained model, the camera and android studio. Additionally, the training of the model was challenging because the platform that was used is not extremely sensitive to subtle differences. The data set that we used had to be carefully selected in order to increase the precision and accuracy of our model.

## Accomplishments that we're proud of

We, as a team, are proud of the fact that we reached our front end goal and finished a complete, marketable application. We are also proud of our machine learning model and android integration. Our final user interface is user friendly to its fullest extent.

## What we learned

We, as a team, learned how integrate a Microsoft custom vision AI and android. We also learned how to create a sleek android user interface for the user. Some team members learned how to step aside and ask other team members for their support and input to their part of the project. This got the foundation of the application running and boosted team morale.

## What's next for Skinmergency

We would like to increase the accuracy of our current model and we would also like to start including and supporting other well known skin conditions. The end goal of Skinmergency is to get real doctor certification in order to improve trust, and reliability of our machine learning model. We would like to possibly take our application to enterprise and become a trusted source of skin condition diagnoses.
## Inspiration

When we joined the hackathon, we began brainstorming about problems in our lives. After discussing some constant struggles in their lives with many friends and family, one response was ultimately shared: health. Interestingly, one of the biggest health concerns that impacts everyone in their lives comes from their *skin*. Even though the skin is the biggest organ in the body and is the first thing everyone notices, it is the most neglected part of the body.

As a result, we decided to create a user-friendly multi-modal model that can discover their skin discomfort through a simple picture. Then, through accessible communication with a dermatologist-like chatbot, they can receive recommendations, such as specific types of sunscreen or over-the-counter medications. Especially for families that struggle with insurance money or finding the time to go and wait for a doctor, it is an accessible way to understand the blemishes that appear on one's skin immediately.

## What it does

The app is a skin-detection model that detects skin diseases through pictures. Through a multi-modal neural network, we attempt to identify the disease through training on thousands of data entries from actual patients. Then, we provide them with information on their disease, recommendations on how to treat their disease (such as using specific SPF sunscreen or over-the-counter medications), and finally, we provide them with their nearest pharmacies and hospitals.

## How we built it

Our project, SkinSkan, was built through a systematic engineering process to create a user-friendly app for early detection of skin conditions. Initially, we researched publicly available datasets that included treatment recommendations for various skin diseases. We implemented a multi-modal neural network model after finding a diverse dataset with almost more than 2000 patients with multiple diseases. Through a combination of convolutional neural networks, ResNet, and feed-forward neural networks, we created a comprehensive model incorporating clinical and image datasets to predict possible skin conditions. Furthermore, to make customer interaction seamless, we implemented a chatbot using GPT 4o from Open API to provide users with accurate and tailored medical recommendations. By developing a robust multi-modal model capable of diagnosing skin conditions from images and user-provided symptoms, we make strides in making personalized medicine a reality.

## Challenges we ran into

The first challenge we faced was finding the appropriate data. Most of the data we encountered needed to be more comprehensive and include recommendations for skin diseases. The data we ultimately used was from Google Cloud, which included the dermatology and weighted dermatology labels. We also encountered overfitting on the training set. Thus, we experimented with the number of epochs, cropped the input images, and used ResNet layers to improve accuracy. We finally chose the most ideal epoch by plotting loss vs. epoch and the accuracy vs. epoch graphs. Another challenge included utilizing the free Google Colab TPU, which we were able to resolve this issue by switching from different devices. Last but not least, we had problems with our chatbot outputting random texts that tended to hallucinate based on specific responses. We fixed this by grounding its output in the information that the user gave.

## Accomplishments that we're proud of

We are all proud of the model we trained and put together, as this project had many moving parts. This experience has had its fair share of learning moments and pivoting directions. However, through a great deal of discussions and talking about exactly how we can adequately address our issue and support each other, we came up with a solution. Additionally, in the past 24 hours, we've learned a lot about learning quickly on our feet and moving forward. Last but not least, we've all bonded so much with each other through these past 24 hours. We've all seen each other struggle and grow; this experience has just been gratifying.

## What we learned

One of the aspects we learned from this experience was how to use prompt engineering effectively and ground an AI model and user information. We also learned how to incorporate multi-modal data to be fed into a generalized convolutional and feed forward neural network. In general, we just had more hands-on experience working with RESTful API. Overall, this experience was incredible. Not only did we elevate our knowledge and hands-on experience in building a comprehensive model as SkinSkan, we were able to solve a real world problem. From learning more about the intricate heterogeneities of various skin conditions to skincare recommendations, we were able to utilize our app on our own and several of our friends' skin using a simple smartphone camera to validate the performance of the model. It’s so gratifying to see the work that we’ve built being put into use and benefiting people.

## What's next for SkinSkan

We are incredibly excited for the future of SkinSkan. By expanding the model to incorporate more minute details of the skin and detect more subtle and milder conditions, SkinSkan will be able to help hundreds of people detect conditions that they may have ignored. Furthermore, by incorporating medical and family history alongside genetic background, our model could be a viable tool that hospitals around the world could use to direct them to the right treatment plan. Lastly, in the future, we hope to form partnerships with skincare and dermatology companies to expand the accessibility of our services for people of all backgrounds.