## Inspiration

Plotgen drew inspiration from the storytellers and artists who excite, move and thrill us with the magic that can be found in their work. We wanted to empower others with the same kind of magic that is now realizable with AI developments like LLMs and Diffusion Models. Seeing how new tools from companies like BentoML and Cohere make it easy for developers to build products from these developments, we knew we had to do something in the space of generative AI.

## What it does

That short story or comic you've always wanted to make? Plotgen can make that dream come true. Given a prompt of a story of interest and several descriptive words, Plotgen generates an entire story with both text and images.

## How we built it

Our frontend was ideated and designed in Figma, and implemented with Next.js and Tailwind CSS. The backend utilizes Cohere's LLM Generate API and a BentoML-powered Stable Diffusion API.

## Challenges we ran into

* Working with inconsistent data from various API sources and building a well-performing pipeline for our use case
* Not being able to access a single GPU strong enough for our image generation models on any of the cloud platforms

## Accomplishments that we're proud of

* Improving the performance of our image model output from unrecognizable faces and subjects to high-quality character art through testing different base models and leveraging LoRAs
* Designers: Successfully implementing new design tools (Spline) & learning CSS to implement the frontend

## What we learned

It was really empowering for us to see how generative AI is super accessible thanks to the development of easy-to-use AI platforms and open-source, but also humbling to experience all the trial and error, bug squashing and uncertainty associated with exploring new technologies.

## What's next for Plotgen

While we accomplished a lot (and far more than what seemed possible at many points in the last 48 hours), there's a lot that we still have to do to deliver on our promise of bringing out the stories that lie within all of us. While our image generation models bring words and characters to life, we would love to do more with them and build multi-panel storylines full of dialogue, rich world-building details and consistent characters fine-tuned to our users' vision for the worlds they're creating.
## **Inspiration:**

Our inspiration stemmed from the realization that the pinnacle of innovation occurs at the intersection of deep curiosity and an expansive space to explore one's imagination. Recognizing the barriers faced by learners—particularly their inability to gain real-time, personalized, and contextualized support—we envisioned a solution that would empower anyone, anywhere to seamlessly pursue their inherent curiosity and desire to learn.

## **What it does:**

Our platform is a revolutionary step forward in the realm of AI-assisted learning. It integrates advanced AI technologies with intuitive human-computer interactions to enhance the context a generative AI model can work within. By analyzing screen content—be it text, graphics, or diagrams—and amalgamating it with the user's audio explanation, our platform grasps a nuanced understanding of the user's specific pain points. Imagine a learner pointing at a perplexing diagram while voicing out their doubts; our system swiftly responds by offering immediate clarifications, both verbally and with on-screen annotations.

## **How we built it**:

We architected a Flask-based backend, creating RESTful APIs to seamlessly interface with user input and machine learning models. Integration of Google's Speech-to-Text enabled the transcription of users' learning preferences, and the incorporation of the Mathpix API facilitated image content extraction. Harnessing the prowess of the GPT-4 model, we've been able to produce contextually rich textual and audio feedback based on captured screen content and stored user data. For frontend fluidity, audio responses were encoded into base64 format, ensuring efficient playback without unnecessary re-renders.

## **Challenges we ran into**:

Scaling the model to accommodate diverse learning scenarios, especially in the broad fields of maths and chemistry, was a notable challenge. Ensuring the accuracy of content extraction and effectively translating that into meaningful AI feedback required meticulous fine-tuning.

## **Accomplishments that we're proud of**:

Successfully building a digital platform that not only deciphers image and audio content but also produces high utility, real-time feedback stands out as a paramount achievement. This platform has the potential to revolutionize how learners interact with digital content, breaking down barriers of confusion in real-time. One of the aspects of our implementation that separates us from other approaches is that we allow the user to perform ICL (In Context Learning), a feature that not many large language models don't allow the user to do seamlessly.

## **What we learned**:

We learned the immense value of integrating multiple AI technologies for a holistic user experience. The project also reinforced the importance of continuous feedback loops in learning and the transformative potential of merging generative AI models with real-time user input.
## Inspiration

We've all wished that we would've talked more to people in person when we had the chance - whether you need them to answer an urgent question, your schedules are too busy for frequent phone calls, or your loved one has passed on, 2nd Chance was built to let you talk to a loved one in real time. Created with the next generation of senior citizens in mind, we wanted to take advantage of the abundance of video and text records people will have of each other while implementing senior-friendly UI. We were excited to use new APIs and learn vector embedding techniques to accomplish our goal of simulating an individual based on their records.

## What it does

2nd Chance first requires a user to submit audio and/or text records of their loved one to get a sense of how they talk. Then, taking either a written or spoken input from the user, the software processes the input and compares it to the records on hand with vector embedding. If the user's input is spoken, it uses OpenAI's Whisper API to transcribe the message into written text. Using ChatGPT to synthesize the existing responses in the records, the software predicts a response to the user's entry. If the user's entry was spoken, we reproduce their loved one's voice using ElevenLab's API so they can speak their response back to the user.

## How we built it

The frontend and backend of our web application was built with Next.js and Convex, respectively. We specifically took advantage of Convex’s Vector Search, Server Functions, and Database features. Our AI simulation was powered by OpenAI’s Chat-GPT model and Vector Embeddings (for Vector Search) were provided by OpenAI’s embedding model. We use Gradio for our speech-to-speech interface, Whisper for audio transcription, and ElevenLabs for voice cloning. Finally, we used Tailwind CSS and Flowbite for overall styling.

## Challenges we ran into

Our decision to use Next.js and Convex caused some difficulty when we attempted to integrate our speech-to-speech software. We were required to implement that software in separate Python files, but then we lost access to our Vector Search software (through Convex). In the end, both speech-to-speech and text-to-text software were integrated onto the web application with Vector Search.

## Accomplishments that we're proud of

We're proud of being able to produce an audio response in the voice of the user's loved one by using ElevenLab's API. It was exciting to get to implement a variety of APIs we hadn't used before to improve our functionality. Another accomplishment was getting the vector embedding to work to help generate responses specific to the user. We’re primarily proud of getting to help people from multiple backgrounds in their day to day lives.

## What we learned

We learned about implementing APIs using their documentations, as well as seeing how many APIs are out there and the value of open source code. We also learned that we should plan out all the technologies we want to use so that we can ensure they can be integrated properly. Lastly, we saw how much we could code we could learn and put together in the span of 36 hours.

## What's next for 2nd Chance

We want to continue to improve the integration of the AI speech-to-speech technology with the front-end design of our website to increase usability and design. We also want to ensure that our user’s text logs and/or audio recordings that they submit as reference points are well encrypted to protect their privacy. It would be nice to also increase functionality for non-English speakers to make our technology more accessible for diverse populations.