## Inspiration

Cliff is dyslexic, so reading is difficult and slow for him and makes school really difficult.
But, he loves books and listens to 100+ audiobooks/yr. However, most books don't have an audiobook, especially not textbooks for schools, and articles that are passed out in class. This is an issue not only for the 160M people in the developed world with dyslexia but also for the 250M people with low vision acuity.
After moving to the U.S. at age 13, Cliff also needed something to help him translate assignments he didn't understand in school.
Most people become nearsighted as they get older, but often don't have their glasses with them. This makes it hard to read forms when needed. Being able to listen instead of reading is a really effective solution here.

## What it does

Audiobook maker allows a user to scan a physical book with their phone to produced a digital copy that can be played as an audiobook instantaneously in whatever language they choose. It also lets you read the book with text at whatever size you like to help people who have low vision acuity or are missing their glasses.

## How we built it

In Swift and iOS using Google ML and a few clever algorithms we developed to produce high-quality scanning, and high quality reading with low processing time.

## Challenges we ran into

We had to redesign a lot of the features to make the app user experience flow well and to allow the processing to happen fast enough.

## Accomplishments that we're proud of

We reduced the time it took to scan a book by 15X after one design iteration and reduced the processing time it took to OCR (Optical Character Recognition) the book from an hour plus, to instantaneously using an algorithm we built.
We allow the user to have audiobooks on their phone, in multiple languages, that take up virtually no space on the phone.

## What we learned

How to work with Google ML, how to work around OCR processing time. How to suffer through git Xcode Storyboard merge conflicts, how to use Amazon's AWS/Alexa's machine learning platform.

## What's next for Audiobook Maker

Deployment and use across the world by people who have Dyslexia or Low vision acuity, who are learning a new language or who just don't have their reading glasses but still want to function. We envision our app being used primarily for education in schools - specifically schools that have low-income populations who can't afford to buy multiple of books or audiobooks in multiple languages and formats.

## Treehack themes

treehacks education Verticle > personalization > learning styles (build a learning platform, tailored to the learning styles of auditory learners) - I'm an auditory learner, I've dreamed a tool like this since the time I was 8 years old and struggling to learn to read. I'm so excited that now it exists and every student with dyslexia or a learning difference will have access to it.

treehacks education Verticle > personalization > multilingual education ( English as a second-language students often get overlooked, Are there ways to leverage technology to create more open, multilingual classrooms?) Our software allows any book to become polylingual.

treehacks education Verticle > accessibility > refugee education (What are ways technology can be used to bring content and make education accessible to refugees? How can we make the transition to education in a new country smoother?) - Make it so they can listen to material in their mother tongue if needed. or have a voice read along with them in English. Make it so that they can carry their books wherever they go by scanning a book once and then having it for life.

treehacks education Verticle >language & literacy > mobile apps for English literacy (How can you build mobile apps to increase English fluency and literacy amongst students and adults?) -One of the best ways to learn how to read is to listen to someone else doing it and to follow yourself. Audiobook maker lets you do that. From a practical perspective - learning how to read is hard and it is difficult for an adult learning a new language to achieve proficiency and a high reading speed. To bridge that gap Audiobook Maker makes sure that every person can and understand and learn from any text they encounter.

treehacks education Verticle >language & literacy > in-person learning (many people want to learn second languages) - Audiobook maker allows users to live in a foreign countrys and understand more of what is going on. It allows users to challenge themselves to read or listen to more of their daily work in the language they are trying to learn, and it can help users understand while they studying a foreign language in the case that the meaning of text in a book or elsewhere is not clear.

We worked a lot with Google ML and Amazon AWS.
## 🌟**Inspiration**

There are over **7.2 million** people in the U.S. who are legally blind, many of whom rely on others to help them navigate and understand their environment. While technology holds the promise of increased independence, current solutions for the visually impaired often fall short—either lacking accessibility features like text-to-speech or offering overly complex interfaces.

Optica was born out of a desire to bridge **this gap**. Our app empowers visually impaired individuals by giving them a simple, intuitive tool to perceive the world independently. Through clear, human-like descriptions of their surroundings, Optica provides not just information, but confidence, autonomy, and a deeper connection to their environment.

## 🛠️ **What it does**

Optica transforms a smartphone into a tool of empowerment for the visually impaired, enabling users to independently understand their surroundings. With the press of a button, users receive clear, succinct, vivid audio descriptions of what the phone’s camera captures. Optica doesn’t just list objects; it paints a picture—communicating the relationships between objects and creating a true sense of place. Optica enables its users to engage with their environment without outside assistance.

## 🧱 **How we built it**

We developed Optica using the ML Kit Object Detection API, which enabled us to identify and classify objects in real-time. These object classifications were then fed into a custom Large Language Model (LLM) powered by TuneStudio and Cerebras, which we trained to generate coherent, natural-language descriptions. The output from this LLM was integrated with Google Cloud’s text-to-speech API to provide users with real-time audio feedback. Throughout development, we maintained a user-first mindset, ensuring that the interface was intuitive and fully accessible.

## ⚔️ **Challenges we ran into**

Developing Optica presented numerous technical and logistical challenges, particularly when it came to integrating various cutting-edge technologies. Deploying our object detection model in Android Studio took longer than anticipated, which limited the time we had to refine other components.

Communicating between our computer vision model and TuneStudio’s LLM proved to be complex, requiring us to overcome issues with API integration and SDK compatibility. Additionally, managing the project across GitHub repositories introduced git-related challenges, particularly when merging contributions from different team members.

However, these difficulties only strengthened our resolve and pushed us to learn new skills—especially in debugging, collaboration, and working across frameworks. Mentors played a crucial role in helping us push through these roadblocks, and the experience has made us better engineers and problem solvers!

## 🎖️ **Our Accomplishments**

We are incredibly proud of our **integration of computer vision and natural language processing**, a combination that allows Optica to go beyond standard object recognition! Starting from a basic CV-based idea, we pushed the boundaries by incorporating an LLM to enhance the descriptions and truly serve the visually impaired community. None of us had experience with these APIs and learned so much on this journey!

Our ability to bring together these powerful technologies to create a tool that can have a tangible, positive impact on people’s lives is an accomplishment we hold in high regard. Successfully deploying this onto a user-friendly platform was a milestone we are excited about.

## 📖 **What we learned**

Although we might have learned new languages, APIs, and git commands on a technical level, the lessons we've learned **go beyond the pages**:

* Setbacks are an inevitable part of the creative process, and staying adaptable allows you to turn challenges into opportunities!
* Starting without all the answers taught us that taking the first step is crucial for personal and project development. We learned to not get ahead of ourselves and take it slow!
* Reaching out for help from our mentors showed us the power of collaboration and shared knowledge. We would like to specifically mention Nifaseth and Harsh Deep for their help!

## ⏭️ **What's next for Optica**

We plan to continually enhance the app by improving the accuracy and breadth of the image classification model, training it on more diverse datasets that include non-conventional settings and real-world complexity. Additionally, we aim to incorporate advanced depth sensing with Google AR’s depth API to provide even more nuanced scene descriptions. On the accessibility front, we will refine the voice activation and gesture-based navigation to make the app even more intuitive. We also look forward to partnering with organizations and sponsors, like Cerebras and TuneStudio, to ensure that **Optica continues to push the boundaries of AI for social good**, helping us realize our vision of full independence for the visually impaired.
## **Inspiration:**

Our inspiration stemmed from the realization that the pinnacle of innovation occurs at the intersection of deep curiosity and an expansive space to explore one's imagination. Recognizing the barriers faced by learners—particularly their inability to gain real-time, personalized, and contextualized support—we envisioned a solution that would empower anyone, anywhere to seamlessly pursue their inherent curiosity and desire to learn.

## **What it does:**

Our platform is a revolutionary step forward in the realm of AI-assisted learning. It integrates advanced AI technologies with intuitive human-computer interactions to enhance the context a generative AI model can work within. By analyzing screen content—be it text, graphics, or diagrams—and amalgamating it with the user's audio explanation, our platform grasps a nuanced understanding of the user's specific pain points. Imagine a learner pointing at a perplexing diagram while voicing out their doubts; our system swiftly responds by offering immediate clarifications, both verbally and with on-screen annotations.

## **How we built it**:

We architected a Flask-based backend, creating RESTful APIs to seamlessly interface with user input and machine learning models. Integration of Google's Speech-to-Text enabled the transcription of users' learning preferences, and the incorporation of the Mathpix API facilitated image content extraction. Harnessing the prowess of the GPT-4 model, we've been able to produce contextually rich textual and audio feedback based on captured screen content and stored user data. For frontend fluidity, audio responses were encoded into base64 format, ensuring efficient playback without unnecessary re-renders.

## **Challenges we ran into**:

Scaling the model to accommodate diverse learning scenarios, especially in the broad fields of maths and chemistry, was a notable challenge. Ensuring the accuracy of content extraction and effectively translating that into meaningful AI feedback required meticulous fine-tuning.

## **Accomplishments that we're proud of**:

Successfully building a digital platform that not only deciphers image and audio content but also produces high utility, real-time feedback stands out as a paramount achievement. This platform has the potential to revolutionize how learners interact with digital content, breaking down barriers of confusion in real-time. One of the aspects of our implementation that separates us from other approaches is that we allow the user to perform ICL (In Context Learning), a feature that not many large language models don't allow the user to do seamlessly.

## **What we learned**:

We learned the immense value of integrating multiple AI technologies for a holistic user experience. The project also reinforced the importance of continuous feedback loops in learning and the transformative potential of merging generative AI models with real-time user input.