## 💭 Motivation

Medical images play a crucial role in training artificial intelligence (AI) algorithms for various diagnostic tasks. However, limited access to large-scale datasets poses a challenge in developing accurate and robust models. Recent advances in latent diffusion models have demonstrated the generation of novel, realistic images using text prompts. Generating high-fidelity medical images could potentially solve the medical data gap. We present an approach for generating synthetic chest X-rays (CXRs) with domain adaptation of Stable Diffusion (SD), a state-of-the-art latent diffusion model for text-to-image generation. We also present a method for removal of text and artifacts from the resulting synthetic images, and evaluated images with two expert radiologists.

## 🔨 Methods

To generate novel CXRs, we fine-tuned the SD v2-1-base model for 10 epochs on an Intel® Max Series GPU, using low-rank adaptation. We used a dataset of 32,532 CXRs paired with clinical reports, a proprietary dataset obtained in a research partnership with Segmed. We then performed GPU inference and generated 1,000 synthetic CXRs using classifier-free guidance of 4, and 75 inference steps.

We implemented a text removal step on images, to eliminate textual artifacts on the generated images. This pipeline leverages text segmentation using Meta’s Segment Anything Model, and inpainting with Stability.AI’s Stable Diffusion 2. The inference is run on an Intel® Max Series GPU and can be executed through a Gradio web application.

## 📈 Evaluation and Results

1. The synthetic images showed quantitative concordance with real CXRs. The Fréchet Inception Distance between the synthetic and real dataset was 68.4, calculated using InceptionV3 pretrained on ImageNet.
2. A random sample of 50 generated CXRs and 50 real CXRs was shuffled, given to a board-certified radiologist with >10 years of clinical experience, and labeled as either "synthetic" or "real". The reader had an accuracy of 0.81, sensitivity 0.86, and specificity 0.76. **Notably, Reader 1 misidentified 7 out of 50 synthetic images as real images.**
3. A random sample of 100 generated CXRs was examined for overall quality (anatomy, style, artifacts) by the same radiologist expert. Some feedback, both positive and negative, is included below in reference to the synthetic X-rays:

>
> A few [synthetic images] stand out due to their sharpness and defined details, making them more closely resemble real X-rays.
>
>
> The rib cage and heart have a "box-y" shape. However, the lung parenchyma looks good.
>
>
> Looks good, except for the ECG marker with a weird position (also the only one).
>
>
> Costo-phrenic angles too deep, esp on the left.
>
>
>

**In total, there were 36 positively scoring samples that were deemed by the radiologist to be comparable to real X-rays, in both anatomy and appearance.** Of the remainder, many were labeled as moderate quality, with the only identified downside being the presence of text artifacts. In the majority of these cases, the AI-enabled text removal is able to rectify this, creating a clean synthetic CXR.

## 🏆 Conclusion

Our pipeline is able to generate synthetic CXRs that resemble real CXRs, with further improvement in performance to be expected as we scale up the training time and data. The proposed method holds promise for augmenting limited medical image datasets, facilitating the development of robust ML algorithms in chest radiography. The ability to generate realistic synthetic CXRs has significant implications for the field of medical imaging. By augmenting limited datasets, our approach can improve training ML algorithms for diagnostic tasks, holding tremendous potential for advancing healthcare outcomes.
## Inspiration

While searching for ideas, our team came across large datasets that needed more effectively classified and were challenging to make predictions based on. Upon examining these datasets, we observed that many could be cleaned and maintained to create models capable of making profound predictions. Following this initial idea, we conceived the idea of developing a website designed to grow exponentially over time, continuously improving its accuracy as we update the models to accommodate expanding datasets. We started by addressing urgent human health issues, such as skin cancer. We developed the website to predict the likelihood of an individual having it, with the support of medical validators, namely doctors.

## What it does

* The system comprises the following components:
* A general user interface in the app, where users can upload images for verification.
* A cloud-based machine learning algorithm.
* A validator interface within the app for verifying data (in this case, images) that produce positive results, which can be either true positives or false positives.
The workflow involves a continuous data exchange between the general user and the machine learning algorithm to detect anomalies. When a trained outcome is predicted, a notification is sent to the validator, who then assesses whether it's a false positive or an accurate positive prediction. If it's a true positive, the validator verifies it and updates it in the positive dataset. If it's a false positive, the validator does not verify it, and it is not added to the positive dataset.

## How it works

* User side: Users can upload images to the model to receive predictions.
* The model evaluates the image.
* The result is returned to the user.
* If the result is positive, the app requests the general user's permission to transfer the image to a validator.
* Validator side: Upon the upload of images and data, the validator assesses the data's quality. After 100 uploads, the validator decides whether to approve the data. Approved data is seamlessly integrated into the dataset, resulting in an automatic update of the AI model.
With the app's help, we can harness the power of big data to develop highly accurate predictive models.

## Building the system

We attempted to create a website using the Reflex development environment to upload images and establish communication with validators to verify them. To accomplish this, we employed a Convolutional Neural Network (CNN) model as our machine learning algorithm, which enabled us to process the continuously incoming data and classify it into "normal" or "abnormal" states. If the data exceeded a threshold for "abnormal" occurrences, we directed it to the validator for verification. Once the validator confirmed the images, we harnessed the capabilities of a database and AI from MindsDB to store and categorize these images. Our models will undergo regular updates using MindsDB's power to enhance accuracy. Consequently, this approach will deliver a more precise and practical prediction application, especially when dealing with large datasets.

## Example use case

Consider a dermatologist using this service to develop and maintain an AI-powered skin cancer detection system. The dermatologist can upload images of skin lesions along with their corresponding diagnoses. Subsequently, the service will train a model for skin cancer detection. The dermatologist can then employ this model to expedite and improve the accuracy of diagnosing new patients.

## Challenges

We faced various challenges and issues throughout this project. Adopting new technologies, such as Reflex and MindsDB, required us to grasp and implement their syntax. Nonetheless, these efforts will yield a more efficient website than current technologies.

The second issue we faced was the limited size of our dataset, which restricted our ability to measure accuracy effectively. In the future, it is advisable to test the model with diverse datasets, explore sequential algorithms, and optimize layers and performance. In the current state, lacking a live information feed and a comprehensive dataset, it is challenging to determine issues related to overfitting or underfitting.

# Future applications

In the next phase of our remote project, we have several key objectives:

* We aim to optimize the system's information processing speed when called upon.
* We plan to expand our network of validators and acquire more datasets to enhance the accuracy of our models.
* We intend to enable the system to automatically generate reports and provide personalized recommendations based on the models.
# giphyFaceSwap

Replace faces in GIFs with your own.
Make sure to check out some snapshots of GIFs made with **GIPHY Face Swap** in the slideshow above.

## Inspiration

GIFs are supposed to be fun and social. Enabling users to customize any of the GIFs from GIPHY's expansive library makes them even more fun and the ability to swap faces with friends or celebrities makes them even more social. Generate a face-swapped GIF and download or share!

## What it Does

Three steps:

* Use the GIPHY API Search Bar to find your favorite GIF with a face in it
* Upload a photo from your computer (selfies work best here, friends)
* Press the face swap button to see the original animated GIF, but now animated with your face on it!

## How I Built It

The front-end of the website is built with HTML, CSS, and JS. I also utilize jquery to interface with the GIPHY API that searches the GIPHY database. The back-end is coded entirely in Python, utilizing a software called OpenCV to analyze the facial structures in the images.
GIFs are unique because they are low-resolution and contain few frames, allowing OpenCV to process the GIF frame-by-frame, creating a custom fit for each frame and taking into account the slight orientation, lighting changes, skin color differences, and size discrepancies between the two faces. The result is an extremely fast processing time and an utterly smooth mask.

## Challenges, Accomplishments, and Learning

I started this project about 12 hours before it was due after my team left the competition. I started from scratch with my own idea and very little web-dev experience. This weekend I learned how to use various JS queries, setup a local server, interface with flask, and access web APIs with the help of the amazing PennApps mentors from Qualtrics.

## What's Next

After the amazing feedback received from testers, I aim to take GIPHY FaceSwap to mobile. This may be a Facebook Messenger app, Android App, iOS App, or all three! This hackathon was an amazing proof-of-concept that users not only want to have GIPHY FaceSwap on their phones, but that they would even be willing to pay for it. I'm looking forward to developing my mobile coding skills and bringing my hack to the world!