Our project aims to leverage computer vision technology to make a contribution to neuroscience research. We were motivated to address the challenges in studying neurite growth dynamics in neurons, a fundamental aspect of neuroscience research. These cells play a crucial role in understanding neurodegeneration, neurodevelopmental disorders, neuron function, and differentiation, making them vital for advancing our understanding of diseases like Alzheimer's, Parkinson's, ASD, and more.

## What we built

NeuraLIVE is a bioinformatic tool designed to streamline and enhance the process of analyzing experiment data on cultured neurons. It employs computer vision techniques to analyze brightfield microscopy images of these cells. Specifically, NeuraLIVE quantifies key aspects of neuron development:

1. **Cell Growth:** NeuraLIVE determines whether the cells proliferate, a critical metric in various experiments.
2. **Cell Health:** The program measures how healthy the cells are by comparing the changing cell shapes over time.
3. **Neuron Morphogenesis:** It identifies sharpness in cell shape and thin, ridge-like objects that represent neurite growth and neuron morphogenesis-- essential processes for understanding the cell's ability to function as neurons.

We have developed an efficient and automated pipeline that can rapidly process and analyze large datasets of brightfield microscopy images. NeuraLIVE not only streamlines the analysis process but also allows analysis of experimental data in a high-throughput matter, truly integrating life science research with the booming field of big data analytics.

## How we built it

We engineered NeuraLIVE by breaking it down into three fundamental tasks, each addressing a critical aspect of neuron analysis:

1: Neuron Identification - Utilizing Watershed Algorithm

The initial task is to analyze input images and determine the number of individual cells present. We implemented the Watershed Algorithm to segment and separate adjacent cell regions, which allows us to identify and quantify the cells in the images.

2: Neuron Classification - Identifying Round and Hypertrophic Cells

Task 2 was vital for distinguishing between round, unadhered cells, and spread-out, adhered cells. We employed contour plots and threshold filters to determine the shape of each cell in the images by distinguishing the number of edges in its contour. This classification not only allowed us to categorize cells but also provided valuable insights into their behavior. Round cells are often indicative of unadhered, unhealthy, cells, while spread-out cells signify adhesion and cell health. Finally, we used the numbers of the two types of cells to calculate the percentage of healthy cells in a given image.

3: Quantifying Neuriteness - Utilizing Ridge Detection Algorithm

The final task, Quantifying Neuriteness, addressed a crucial question: how much sharp, neurite-like structures are present in the cells? To accomplish this, we integrated the Ridge Detection Algorithm. This algorithm identifies and counts the number of straight, linear, and sharp objects within the images, which corresponded to neurogenesis. Quantification of neurite outgrowth is central to understanding neurodevelopment and neurodegenerative processes.

4: Front End Website
To upload data sets and analyze the pictures we built a website using Flask as framework and HTML for the UI.

## Challenges we ran into

Developing NeuraLIVE presented several challenges, including:

*Data Parsing:* We started off by taking all the images in the dataset as one culture and only found out later that they are actually from four different cultures. Thus, we had to restart the sorting process of the images to consider dividing the whole dataset into four cultures when plotting. This takes a while to implement and debug since the images are in random order, and parsing to find each image file's culture involves many unfamiliar syntax.

*Ridge Detection:* Quantifying neuriteness was extremely challenging--brightfield images are known for their difficulty in being analyzed, especially the sharper, thinner objects that do not reflect much light. In the beginning, the Ridge Detection Algorithm we implemented frequently identify non-neurite structures as neurites. We investigated the implementation of multiple ridge filters and explored options from various packages, and eventually choose the Meijuring filter with a customized greyscale threshold value to isolate the neurites from the cell bodies.

## Accomplishments that we're proud of

*Caroline:* I'm most proud of cooperating so perfectly with my team in these 48 hours and finishing such an innovative project.
*Georges:\_I am immensely proud of the technical innovation that NeuraLIVE represents. We successfully applied computer vision techniques to tackle a significant problem in neuroscience research. This project isn't just about code; it's about leveraging technology to advance our understanding of critical issues like neurodegeneration, neurodevelopmental disorders, and neuron function. The implication that NeuraLIVE can have on these fields is a testament to our dedication.
\_Millie*: I am super proud of how supportive and collaborative we are to each other. Team work makes the dream work!
*Steve*: As a student in both life science and computer science, it has always been my dream to carry out a project that combines my two fields of passion and make something that can potentially be useful to the research world. I believe NeuraLIVE, although being a product built within 48 hours of time, contains ideas and implementations that can be truly useful to the research field out there. I am so, so proud of my teammates, and thankful for theirs and our efforts in making this thing a reality along the way.

## What we learned

*Caroline*: I was exposed to and learned a lot about the intersection of computer vision and biology through this project.
*Millie*: I learned about neurons and many different useful python libraries for computer vision.
*George*: As a student who knows much more about business than biology, this is something I haven't touched much before, but I absolutely love it! Hoping to maybe even take a biology class in the future.
*Steve*: I learned so many useful image processing and computer vision libraries in Python, and learned about front-end development--something I have not worked with before!

## What's next for NeuraLIVE

We plan to incorporate machine learning in future implementations, aiming to improve accuracy in object identification and segmentation. We also plan on optimizing several pixel-thresholding algorithm to improve the general runtime of our program.
Discord usernames: Hexular#5020, sophaiya#3995, photonmz#4461

Check out our mobile prototype [demo video](https://drive.google.com/file/d/1U78Z40JO3vEEElFg_6SBHcMw-Go1PFJT/view?usp=sharing) and [prototype](https://www.figma.com/proto/XcamQs8aNpZ8veXBw6ukGn/Hack-the-6ix?node-id=264%3A6785&scaling=scale-down&page-id=14%3A116&starting-point-node-id=264%3A6785&show-proto-sidebar=1)!

## Inspiration

As avid listeners and lovers of music, we couldn't help but notice that our playlists have inherent limitations. Playlists force the user to manually specify all the songs they want to listen to, a time-consuming task, and can only be experienced in different orders; the songs themselves stay fixed, meaning it's really easy to get bored of a playlist, a thing that I'm sure we've all experienced at least once.

While apps like Spotify, Apple Music and YouTube Music do allow for automated playlists, e.g. Spotify's Daily Mix, they have several downsides. For example, you as the user have no control over these recommendations; it's very common for the Daily Mix to deliver a song that's a complete miss. Another problem is that you can't impose any constraints on the input songs; I'm sure we all have that one awful song that consistently pops up in the recommended songs list and never goes away.

We wished to create something that would help experienced music listeners organize their preferred songs in an intuitive way, automating away all of the manual work while simultaneously giving the user the power to manually control things if they want even finer-grained recommendations. Thus, we created Playflow.

## What it does

Playflow's killer feature is "playflows": they're the next step up from playlists. Just like how playlists transcended albums by adding more freedom and removing manual work, so too do playflows transcend playlists by adding more freedom and removing manual work.
You can think of a "playflow" like a template of a playlist: it provides ways to generate sequences of songs.
For example, instead of having to manually choose the song "Vexento - Pixel Party" and the song "Tobu - Cloud Nine" and the song "MDK - Press Start" and a million others, you could specify "Give me 30 minutes of Energetic EDM songs", and Playflow will give you all of the options you might have selected and more. Or you could specify "Give me 30 minutes of songs from Vexento". Or you could specify "Give me 10 seconds of phone ringtones", just for kicks. And you can sequence these all in order. Say you're running a hackathon (totally arbitrary example) and you want to play a nice list of songs for the entire duration, but you don't want to spend the multiple hours it would take manually adding songs to make a 24-hour playlist complete with rising and falling action music for submission deadlines. All you need to input is "Give me 20 hours of chill work songs", "Give me 2 hours of energetic work songs", "Give me 1 hour of dramatic songs", and then "Give me 1 hour of prize ceremony songs". Way easier, right? You could even specify "Give me 3 hours of songs like Imagine Dragons - Radioactive". The AI behind the scenes handles this all for you, so you don't have to do any other work, but you can still manually specify "Give me Imagine Dragons - Radioactive" if you really want to desperately listen to it.

## Why's this better than Spotify?

Suppose you're going to go for a long run while listening to music, and you know you're really going to struggle at the end, so you want to listen to motivational songs at the end to help you. If you're doing this with a normal playlist, you would have to manually choose all of your motivational songs at the end of your run, and manually choose a bunch of normal songs to play at the start of your run; easily a 30-minute task, and you have to hope you never get bored of it. With Playflow, all you have to do is tell the app to give you "45 minutes of songs I like" and "15 minutes of motivational music"; 30 seconds and you have your playlist ready to go running with, which will constantly have different songs in it, making sure you're never bored. Playflow grants more control to the listener.

## Technical details

The app uses Progressive Web Application (PWA) technology to increase development velocity and ensure that it works cross-platform on both Android and iOS. The PWA is built using React + Tailwind + Typescript, with React-Router to manage pages and Recoil to help manage state, and hosted on Netlify. The app communicates with a backend server storing data, an Express application running on Debian on Amazon EC2 utilizing LowDB for user data and an Apache Cassandra database instance for efficient ML model queries. The ML model itself was trained by SageMaker, Gensim and TensorFlow using the Word2Vec model, with Jupyter and Seaborn for visualization and K-dimensional tree datastructures (kd-trees) for efficient querying. To enable the use of OAuth2, we use an AWS Lambda deployment using the declarative Serverless framework, utilizing CloudFormation and IAM as well as CloudWatch for logging. And in the center of it all, we use the Spotify API as both a producer of songs, a consumer of song queues, and an OAuth2 client.

## Challenges

We initially attempted to use AWS Keyspaces (a managed deployment of Cassandra) as our ML database, but ran into multiple issues which took us a significant chunk of time to debug.

## What went well

The Spotify API code was extremely straightforward and easy to implement.

## What's next for Playflow

In the future, we plan to integrate Playflow with other music apps such as Apple Music or YouTube Music. We also aspire to link geographical location data and other user data with the app so that AI can help suggest playflows for the user to use depending on their location (e.g. listening to energetic music while at a gym) or time (e.g. listening to calm music during sleep hours).
# Flash Computer Vision®

### Computer Vision for the World

Github: <https://github.com/AidanAbd/MA-3>

Try it Out: <http://flash-cv.com>
## Inspiration

Over the last century, computers have gained superhuman capabilities in computer vision. Unfortunately, these capabilities are not yet empowering everyday people because building an image classifier is still a fairly complicated task.

The easiest tools that currently exist still require a good amount of computing knowledge. A good example is [Google AutoML: Vision](https://cloud.google.com/vision/automl/docs/quickstart) which is regarded as the "simplest" solution and yet requires an extensive knowledge of web skills and some coding ability. We are determined to change that.

We were inspired by talking to farmers in Mexico who wanted to identify ready / diseased crops easily without having to train many workers. Despite the technology existing, their walk of life had not lent them the opportunity to do so. We were also inspired by people in developing countries who want access to the frontier of technology but lack the education to unlock it. While we explored an aspect of computer vision, we are interested in giving individuals the power to use all sorts of ML technologies and believe similar frameworks could be set up for natural language processing as well.

## The product: Flash Computer Vision

### Easy to use Image Classification Builder - The Front-end

Flash Magic is a website with an extremely simple interface. It has one functionality: it takes in a variable amount of image folders and gives back an image classification interface. Once the user uploads the image folders, he simply clicks the Magic Flash™ button. There is a short training process during which the website displays a progress bar. The website then returns an image classifier and a simplistic interface for using it. The user can use the interface (which is directly built on the website) to upload and classify new images. The user never has to worry about any of the “magic” that goes on in the backend.

### Magic Flash™ - The Backend

The front end’s connection to the backend sets up a Train image folder on the server. The name of each folder is the category that the pictures inside of it belong to. The backend will take the folder and transfer it into a CSV file. From this CSV file it creates a [Pytorch.utils.Dataset](https://pytorch.org/docs/stable/_modules/torch/utils/data/dataset.html) Class by inheriting Dataset and overriding three of its methods. When the dataset is ready, the data is augmented using a combination of 6 different augmentations: CenterCrop, ColorJitter, RandomAffine, RandomRotation, and Normalize. By doing those transformations, we ~10x the amount of data that we have for training. Once the data is in a CSV file and has been augmented, we are ready for training.
We import [SqueezeNet](https://pytorch.org/hub/pytorch_vision_squeezenet/) and use transfer learning to adjust it to our categories. What this means is that we erase the last layer of the original net, which was originally trained for ImageNet (1000 categories), and initialize a layer of size equal to the number of categories that the user defined, making sure to accurately match dimensions. We then run back-propagation on the network with all the weights “frozen” in place with exception to the ones in the last layer. As the model is training, it is informing the front-end that progress being made, allowing us to display a progress bar. Once the model converges, the final model is saved into a file that the API can easily call for inference (classification) when the user asks for prediction on new data.

## How we built it

The website was built using Node.js and javascript and it has three main functionalities: allowing the user to upload pictures into folders, sending the picture folders to the backend, making API calls to classify new images once the model is ready.

The backend is built in Python and PyTorch. On top of Python we used torch, torch.nn as nn, torch.optim as optim, torch.optim import lr\_scheduler, numpy as np, torchvision, from torchvision import datasets, models, and transforms, import matplotlib.pyplot as plt, import time, import os, import copy, import json, import sys.

## Accomplishments that we're proud of

Going into this, we were not sure if 36 hours were enough to build this product. The proudest moment of this project was the successful testing round at 1am on Sunday. While we had some machine learning experience on our team, none of us had experience with transfer learning or this sort of web application. At the beginning of our project, we sat down, learned about each task, and then drew a diagram of our project. We are especially proud of this step because coming into the coding portion with a clear API functionality understanding and delegated tasks saved us a lot of time and helped us integrate the final product.

## Obstacles we overcame and what we learned

Machine learning models are often finicky in their training patterns. Because our application allows users with little experience, we had to come up with a robust training pipeline. Designing this pipeline took a lot of thought and a few failures before we converged on a few data augmentation techniques that do the trick. After this hurdle, integrating a deep learning backend with the website interface was quite challenging, as the training pipeline requires very specific labels and file structure. Iterating the website to reflect the rigid protocol without over complicating the interface was thus a challenge. We learned a ton over this weekend. Firstly, getting the transfer learning to work was enlightening as freezing parts of the network and writing a functional training loop for specific layers required diving deep into the pytorch API. Secondly, the human-design aspect was a really interesting learning opportunity as we had to come up with the right line between abstraction and functionality. Finally, and perhaps most importantly, constantly meeting and syncing code taught us the importance of keeping a team on the same page at all times.

## What's next for Flash Computer Vision

### Application companion + Machine Learning on the Edge

We want to build a companion app with the same functionality as the website. The companion app would be even more powerful than the website because it would have the ability to quantize models (compression for ml) and to transfer them into TensorFlow Lite so that **models can be stored and used within the device.** This would especially benefit people in developing countries, where they sometimes cannot depend on having a cellular connection.

### Charge to use

We want to make a payment system within the website so that we can scale without worrying about computational cost. We do not want to make a business model out of charging per API call, as we do not believe this will pave a path forward for rapid adoption. **We want users to own their models, this will be our competitive differentiator.** We intentionally used a smaller neural network to reduce hosting and decrease inference time. Once we compress our already small models, this vision can be fully realized as we will not have to host anything, but rather return a mobile application.