## Inspiration

Throughout the COVID-19 pandemic, there has been an astounding 86% increase in technology usage amongst senior citizens.

However, with feature rich platforms found everywhere on the internet, big walls of text and countless buttons can overwhelm those who are not familiar with technology. Senior folks who are used to face-to-face interactions are now having a difficult time adjusting to doing their everyday tasks online.

## What it does

Using genuine user activity data, the Walkthru chrome extension develops highlighted walk-throughs of the most common use cases. By either selecting an existing use case, or using the voice command feature, Walkthru immediately begins the guidance process from start to end in order to accomplish any task you may have.

## How we built it

Walkthru is built as a Chrome extension that automatically loads the appropriate common actions taken on a website. These common actions are based off user data taken from website heatmap data sources such as Hotjar.

For the voice command feature, we delegate Google Speech-to-Text to match the spoken command to a predefined use case, if one exists. Walkthru even accepts commands in foreign languages, thanks to the Speech-to-Text multilingual functionality.

## Challenges we ran into

Assuming the perspective of senior citizens without being one; often had to overcome our own biases by speaking with senior citizens firsthand.

## Accomplishments that we're proud of

A unique and unparalleled solution to simple yet intelligent guidance for senior citizens that can be scaled as an accessibility tool for other demographics and differently-abled people.

## What's next for Walkthru

Combine Machine Learning with the Hotjar heatmap data to automate the use case guidance creation process, since use cases are currently created manually.
## Inspiration

Our team is made of story lovers. For as long as we can remember, we have been seeking to consume and create stories wherever we go. As we discovered our vastly different reading methodologies due to dyslexia, varying native languages, and differentiated learning experiences, we sought to create a platform that allows all adventurers to immerse themselves in stories.

Journey was created from a desire to enhance users’ reading experiences from the very beginning: childhood. During our ideation process, our research indicated two major findings:
1) Children who were regularly read to showed significantly higher levels of reading comprehension in developing years.
2) Time spent reading decreases as children age because they expect to comprehend harder-to-read stories less. This is especially prevalent when children are transitioning from picture-books to novels.

As we contemplated ways to immerse children in worlds of literature, the idea of integrating AI was presented as a powerful catalyst for engagement. Educators could use this tool to bring stories to life and allow students to clarify their questions first-hand, thus transcending traditional reading methods and innovating the educational experience.

Calling from our personal experiences, our early ideas involved using AI generated text-to-speech to deliver literature to students struggling with dyslexia or English as a second language. However, after many rounds of idea and product development, we are proud to deliver much more than simple text-to-speech. We are proud to have implemented our personal stories into aiding others in developing their own stories as well.

## What it does

Journey enables readers to interact live-time with book characters as they explore stories. Customizable conversations allow users to gain explicit insights regarding plot, settings, and even characters’ opinions of each other. Journey’s interactable characters are unlocked in time with their introduction in book—in other words, readers get to meet characters such as Ron Weasley at the same time as Harry Potter!

Journey’s mythical interface and speech-to-text features create an exciting visual and auditory experience. Watch as settings magically appear as you read, listen to Journey’s storyteller weave your favourite tales, and experience the beauty of a great book.

Journey will break down daunting chunks of text into bite-sized pieces and allow users to enjoy a comprehensive reading experience. By simply uploading a PDF book file, you can start your reading journey!

## How we built it

Development started by fine turning AI prompts based of story context, who the user is roleplaying, and who the character is to perfectly match the kind of response we had in mind. We aimed to generate creative responses that conveyed information relevant up the point in the story the reader is at while mimicking emotions of the character the AI is portraying. An omniscient narrator is also available for questioning by the user. The narrator knows everything about the story and provides and unbiased an objective view on the story. This can be used for clarification or a brief summary of the story.

To gather the material, novels were obtained in pdf form and were parsed through and split into pages. The pages are displayed main application interface, providing the opportunity interactively parse through the pages in the list. The AI intelligently knows where the reader is when selecting a page based on its content; thus, only allowing the user to talk to a character that has been introduced up to that page.

Journey’s visual element was brought to life with the aid of Open AI’s Dall-E, an image generation tool. Each character and page background has a unique generation made by the AI through a simple API call. Each avatar is displayed next their character under the ‘Character Name’ section.

## Challenges we ran into

* Integrating each feature and tool we used into one cohesive product (TTS, character generation, string parsing, character responses, and background art generation)
* Staying conscious of our budget while creating API requests to test our code
* Preventing characters from spoiling later parts of the book or breaking the fourth wall when asking them questions
* Training the AI to tailor responses to emotions based off characters in the book depending on who they’re interacting with
* Matching the different TTS voices OpenAI offers depending on the character they’re narrating as
* Staying hydrated and healthy while racing to meet the 36-hour deadline!

## Accomplishments that we're proud of

* This was Anthony Botticchio, Larry Han, and Claire Hu’s first ever hackathon and Claire’s first-time programming!
* Elements in our Logo and UI/UX were 100% drawn by hand using vector art on Figma! We wanted to follow an RPG game style theme, so custom making everything was the only way to go.
* One of our team members has personally faced learning disabilities, so creating a solution for firsthand problems was incredibly fulfilling. We believe this experience gave us a deeper understanding of the current pain points, allowing us to craft a tailored solution for the problem space we identified.

## What we learned

* Hackathons are not easy…one hour seems to go by in fifteen minutes!
* OpenAI is extremely powerful, however tremendous training is needed to make it robust
* Grouping designs in Figma is essential—otherwise your frames will get very messy very fast!

## What's next for Journey

We hope to continue developing Journey and helping students learn and enjoy their reading experiences. As the problem space we chose to tackle is close to our hearts, we truly believe others struggling with similar experiences deserve the chance to explore wonderful stories in an easier fashion than we have.

Reading is instrumental in the development of children, and we hope that by continuing to develop Journey we can give a generation of children a love for reading!

## References:

The Washington Post. (2015, April 29). Why kids lose interest in reading as they get older. <https://www.washingtonpost.com/news/answer-sheet/wp/2015/04/29/why-kids-lose-interest-in-reading-as-they-get-older/>
Reading Rockets. (n.d.). Why some children have difficulties learning to read. <https://www.readingrockets.org/topics/struggling-readers/articles/why-some-children-have-difficulties-learning-read>
Child Mind Institute. (n.d.). Why is it important to read to your child? <https://childmind.org/article/why-is-it-important-to-read-to-your-child/>
The Ohio State University College of Education and Human Ecology. (n.d.). The importance of reading to kids daily. <https://ehe.osu.edu/news/listing/importance-reading-kids-daily-0>
National Assessment of Educational Progress. (n.d.). The Nation’s Report Card: Reading Achievement. <https://www.nationsreportcard.gov/reading/nation/achievement/?grade=4>
Yale Center for Dyslexia & Creativity. (n.d.). Dyslexia FAQ. <https://dyslexia.yale.edu/dyslexia/dyslexia-faq/>
Cognitive Market Research. (n.d.). English Language Learning Market Report. <https://www.cognitivemarketresearch.com/english-language-learning-market-report>
Government of Canada. (n.d.). Official Languages and Bilingualism Publications: Statistics. <https://www.canada.ca/en/canadian-heritage/services/official-languages-bilingualism/publications/statistics.html>
DoteFL. (n.d.). English Language Statistics. <https://www.dotefl.com/english-language-statistics/>
National Center for Education Statistics. (n.d.). English Learners in Public Schools. <https://nces.ed.gov/programs/coe/indicator/cgf/english-learners>
Statista. (n.d.). Resident population of Canada by age group. <https://www.statista.com/statistics/444868/canada-resident-population-by-age-group/>
Statista. (n.d.). Population of the United States by sex and age. <https://www.statista.com/statistics/241488/population-of-the-us-by-sex-and-age/>
## Inspiration

We wanted to come to Deltahacks not to just create some product but create a product with lasting impact. We choose to tackle an area that we felt was underserved. We decided to create a command portal for people with ALS. Command portal ? You're probably wondering what I'm saying. We'll show you as we go along!

## What it does

We created a command portal that allows people with ALS to communicate with the world. If you know anyone who suffers from ALS then you would also know how they are completely immobilized. We decide to factor the medium for communication between the device and the user with blinking! With this we were able to have users open and close doors, send help messages to a loved one and even open a blink to text translator that would allow the user to communicate with the world in ways which we have never fathomed.

## How we built it

The entire setup consists or 4 main features. The first feature is the blink detection which is powered with 3d vector and spatial mapping. With this technology we are able to map the human eye with the capabilities of any device with a depth sensor. Next up we have the text message feature. In this we see how a user with ALS who is in trouble can blink a certain number of times to trigger a automated message that allows a loved one to know they are in trouble. This is powered by the Twilio API. Next up we have the door open and close feature. This feature allows for users who wish to open doors to blink a certain pattern to trigger a door opening or closing. The final feature is a blink to text translator which is using morse code to identify certain letters and display it on the screen. We feel that with morse code the possibilities are endless for people with ALS to start to move towards a more connected life!

## What's next for BlinkBuddy

We plan to divert in either 1 of 2 possibilities. The first possibility is to work on the computer vision and spatial mapping and increase the accuracy of such or to potentially convert our blink detection to be more accurate using EMGs. EMGs or more commonly known as electromyography are muscle detectors and could be placed in pair with the blink detector CV to identify blink with greater accuracy