## Inspiration

With recent booms in AI development, deepfakes have been getting more and more convincing. Social media is an ideal medium for deepfakes to spread, and can be used to seed misinformation and promote scams. Our goal was to create a system that could be implemented in image/video-based social media platforms like Instagram, TikTok, Reddit, etc. to warn users about potential deepfake content.

## What it does

Our model takes in a video as input and analyzes frames to determine instances of that video appearing on the internet. It then outputs several factors that help determine if a deepfake warning to a user is necessary: URLs corresponding to websites where the video has appeared, dates of publication scraped from websites, previous deepfake IDs (i.e. if the website already mention the words "deepfake"), and similarity scores between the content of the video being examined and previous occurrences of the deepfake. A warning should be sent to the user if content similarity scores between it and very similar videos are low (indicating the video has been tampered with) or if the video has been previously IDed as a deepfake by a different website.

## How we built it

Our project was split into several main steps:

**a) finding web instances of videos similar to the video under investigation**
We used Google Cloud's Cloud Vision API to detect web entities that have content matching the video being examined (including full matching and partial matching images).

**b) scraping date information from potential website matches**
We utilized the htmldate python library to extract original and updated publication dates from website matches.

**c) determining if a website has already identified the video as a deepfake**
We again used Google Cloud's Cloud Vision API to determine if the flags "deepfake" or "fake" appeared in website URLs. If they did, we immediately flagged the video as a possible deepfake.

**d) calculating similarity scores between the contents of the examined video and similar videos**
If no deepfakes flags have been raised by other websites (step c), we use Google Cloud's Speech-to-Text API to acquire transcripts of the original video and similar videos found in step a). We then compare pairs of transcripts using a cosine similarity algorithm written in python to determine how similar the contents of two texts are (common, low-meaning words like "the", "and", "or", etc. are ignored when calculating similarity).

## Challenges we ran into

Neither of us had much experience using Google Cloud, which ended up being a major tool in our project. It took us a while to figure out all the authentication and billing procedures, but it was an extremely useful framework for us once we got it running.

We also found that it was difficult to find a deepfake online that wasn't already IDed as one (to test out our transcript similarity algorithm), so our solution to this was to create our own amusing deepfakes and test it on those.

## Accomplishments that we're proud of

We're proud that our project mitigates an important problem for online communities. While most current deepfake detection uses AI, malignant AI can simply continually improve to counter detection mechanisms. Our project takes an innovative approach that avoids this problem by instead tracking and analyzing the online history of a video (something that the creators of a deepfake video have no control over).

## What we learned

While working on this project, we gained experience in a wide variety of tools that we've never been exposed to before. From Google Cloud to fascinating text analysis algorithms, we got to work with existing frameworks as well as write our own code. We also learned the importance of breaking down a big project into smaller, manageable parts. Once we had organized our workflow into reachable goals, we found that we could delegate tasks to each other and make rapid progress.

## What's next for Deepfake ID

Since our project is (ideally) meant to be integrated with an existing social media app, it's currently a little back-end heavy. We hope to expand this project and get social media platforms onboard to using our deepfake detection method to alert their users when a potential deepfake video begins to spread. Since our method of detection has distinct advantages and disadvantages from existing AI deepfake detection, the two methods can be combined to create an even more powerful deepfake detection mechanism.

Reach us on Discord: **spica19**
## Inspiration

The prevalence of fake news has been on the rise. It has led to the public's inability to receive accurate information and has placed a heightened amount of distrust on the media. With it being easier than ever to propagate and spread information, the line between what is fact and fiction has become blurred in the public sphere. Concerned by this situation, we built a mobile application to detect fake news on its websites and alert people when information is found to be false or unreliable, thereby hopefully bringing about a more informed electorate.

## What it does

enlightN is a mobile browser with built-in functionality to detect fake news and alert users when the information they are reading - on Facebook or Twitter - is either sourced from a website known for disseminating fake news or known to be false itself. The browser highlights which information has been found to be false and provides the user sources to learn more about that particular article.

## How we built it

**Front-end** is built using Swift and Xcode. The app uses Alamofire for HTTP networking, and WebKit for the browser functionality. Alamofire is the only external dependency used by the front end; other than that it's all Apple's SDK's. The webpage HTML is parsed and sent to the backend, and the response is parsed on the front end.

**Back-end** is built using Python, Google App Engine, Microsoft Cognitive Services, HTML, JavaScript, CSS, BeautifulSoup, Hoaxy API, and Snopes Archives. After receiving the whole HTML text from front-end, we scrape texts from Facebook and Twitter posts with the use of the BeautifulSoup module in Python. Using the keywords of the texts by Microsoft Key Phrase Extraction API (which uses Microsoft Office's Natural Language Processing toolkit) as an anchor, we extract relevant information (tags for latent fake news) from both Snopes.com's Database and the results getting back from the hoaxy API and send this information back to the front-end.

**Database** contains about 950 websites that are known for unreliable (e.g. fake/conspiracy/satire) news sources and about 15 well-known trustworthy news source websites.

## Challenges we ran into

One challenge we ran into was with implementing the real-time text search in order to cross-reference article headlines and Tweets with fact-checking websites. Our initial idea was to utilize Google’s ClaimReview feature on their public search, but Google does not have an API for their public search feature and after talking to some of the Google representatives, automating this with a script would not have been feasible. We then decided to implement this feature by utilizing Snopes. Snopes does not have an API to access their article information and loads their webpage dynamically, but we were able to isolate the Snopes’ API call that they use to provide their website with results from an article query. The difficult part of recreating this API call was figuring out the proper way to encode the POST payload and request header information before the HTTP function call.

## Accomplishments that we're proud of

We were able to successfully detect false information from any site after especially handling facebook and twitter. The app works and makes people aware of disinformation in real-time!

## What we learned

We applied APIs that are completely new for us - Snopes’ API, hoaxy API, and Key Phrase Extraction API - in our project within the past 36 hours.

## What's next for enlightN

Building a fully-functional browser and an app which detects false information on any 3rd party app. We also plan to publicize our API as it matures.
## Inspiration

The upcoming election season is predicted to be drowned out by mass influx of fake news. Deepfakes are a new method to impersonate famous figures saying fictional things, and could be particularly influential in the outcome of this and future elections. With international misinformation becoming more common, we wanted to develop a level of protection and reality for users. Facebook's Deepfake Detection Challenge, which aims to crowdsource ideas, inspired us to approach this issue.

## What it does

Our Chrome extension processes the current video the user is watching. The video is first scanned using AWS to identify the politician/celebrity in subject. Knowing the public figure allows the app to choose the model trained by machine learning to identify Deepfakes targeting that specific celebrity. The result of the deep fake analysis are then shown to the user through the chrome extension, allowing the user to see in the moment whether a video might be authentic or forged.

Our Web app offers the same service, and includes a prompt that describes this issue to users. Users are able to upload a link to any video they are concerned may be tampered with, and receive a result regarding the authenticity of the video.

## How we built it

We used the PCA (Principal Component Analysis) to build the model by hand, and broke down the video into one second frames.

Previous research has evaluated the mouths of the deepfake videos, and noticed that these are altered, as are the general facial movements of a person. For example, previous algorithms have looked at mannerisms of politicians and detected when these mannerisms differed in a deepfake video. However, this is computationally a very costly process, and current methods are only utilized in a research settings.

Each frame is then analyzed with six different concavity values that train the model. Testing data is then used to try the model, which was trained in a Linear Kernel (Support Vector Machine). The Chrome extension is done in JS and HTML, and the other algorithms are in Python.

The testing data set is comprised from the user's current browser video, and the training set is composed of Google Images of the celebrity.

## Challenges we ran into

Finding a dataset of DeepFakes large enough to train our model was difficult. We ended up splitting the frames into 70% dataset, and 30% of it is used for testing (all frames are different however). Automatically exporting data from JavaScript to Python was also difficult, as JavaScript can not write on external files.

Therefore, we utilized a server and were able to successfully coordinate the machine learning with our web application and Chrome extension!

## Accomplishments that we're proud of

We are proud to have created our own model and make the ML algorithm work! It was very satisfying to see the PCA clusters, and graph the values into groups within the Support Vector Machine. Furthermore, getting each of the components ie, Flask and Chrome extension to work was gratifying, as we had little prior experience in regards to transferring data between applications.

We are able to successfully determine if a video is a deep fake, and notify the person in real time if they may be viewing tampered content!

## What we learned

We learned how to use AWS and configure the credentials, and SDK's to work. We also learned how to configure and utilize our own machine learning algorithm, and about the dlib/OpenCV python Libraries!

Furthermore, we understood the importance of the misinformation issue, and how it is possible to utilize a conjuction of a machine learning model with an effective user interface to appeal to and attract internet users of all ages and demographics.

## What's next for DeFake

Train the model with more celebrities and get the chrome extension to output whether the videos in your feed are DeepFakes or not as you scroll. Specifically, this would be done by decreasing the run time of the machine learning algorithm in the background. Although the algorithm is not as computationally costly as conventional algorithms created by experts, the run time barrs exact real-time feedback within seconds.

We would also like to use the Facebooks DeepFake datasets when they are released.
Specifically, deepfakes are more likely to become a potent tool of cyber-stalking and bullying in the short term, says Henry Ajder, an analyst at Deeptrace. We hope to utilize larger data bases of more diverse deepfakes outside of celebrity images to also prevent this future threat.