## Inspiration

Education today relies heavily on online platforms and resources, and we all spend vast amount of time on screen everyday. Our inspiration for LensEd stemmed from a deep desire to create a more inclusive online environment for individuals with disabilities, particularly those with conditions like Dyslexia. We were inspired by the idea that technology could be harnessed to make web content more accessible, enabling everyone to have an equal opportunity to engage with digital resources.

## What it does

LensEd is a Google Chrome extension designed with accessibility in mind. It offers a range of features to enhance the online experience for users with disabilities or visual impairments. Here's what LensEd does:

Customizable Display: LensEd allows users to change the background color, font style, and text color on web pages. This customization enables individuals with Dyslexia or other visual challenges to tailor their browsing experience to their specific needs.

Metaphor API Integration: We integrate Metaphor API to create a powerful search engine within LensEd. Users can search for content or find similar websites effortlessly, enhancing their ability to access relevant information efficiently.

## How we built it

LensEd was built with a combination of technologies and tools. The development process involved the following steps:

Conceptualization: We brainstormed and outlined the features and goals of the extension, focusing on accessibility and usability.

Front-End Development: We used HTML, CSS, and JavaScript to create the user interface and implement the customization features. These elements allow users to adjust the visual aspects of websites.

Metaphor API Integration: To implement the powerful search engine, we integrated the Metaphor API into our extension, enabling seamless content discovery.

Testing and Feedback: We conducted extensive testing with users who have disabilities and collected feedback to refine the user experience and ensure accessibility.

Deployment: Once we were satisfied with the functionality and usability, we deployed LensEd as a Google Chrome extension.

## Challenges we ran into

Developing LensEd came with its share of challenges, including:

Accessibility Standards: Ensuring that our extension meets strict accessibility standards is crucial. We need to make sure our color scheme and design provide a seamless experience for users with disabilities.

API Integration: Integrating the Metaphor API was a technically complex task that required extensive documentation reading and testing.

## Accomplishments that we're proud of

Despite the challenges, we're immensely proud of several accomplishments:

Accessibility Impact: LensEd makes the internet more accessible for individuals with disabilities, leveling the playing field and improving their online experience.

Seamless Search: The integration of the Metaphor API has resulted in a powerful search engine that makes content discovery easier and more efficient.

User-Centric Design: Our extension's user-centric design prioritizes the needs of individuals with disabilities, creating a more inclusive web.

## What we learned

Through the development of LensEd, we learned valuable lessons about accessibility, user experience, and the power of technology to create positive change. We gained insights into the challenges faced by individuals with disabilities in their online interactions and the importance of developing solutions that address these challenges.

## What's next for LensEd

Our journey with LensEd is far from over. We have several exciting plans for the future:

Expanded Features: We aim to expand the customization options within LensEd, giving users even more control over their browsing experience. Besides, we are working on adding in more features such as auto scrolling using speech-to-text via Google Cloud API.

Cross-Browser Compatibility: We plan to make LensEd available on multiple browsers to reach a wider audience.

Community Involvement: We want to collaborate with the user community to gather feedback and continually improve LensEd's accessibility and functionality.

Overall, LensEd is an ongoing project with the mission to make the web a more accessible and inclusive place for everyone, and we are excited to continue working towards that goal.
## GitHub

<https://github.com/sina-jamshidi/nwHacks2021-ShareStreet>

## Cloud Infrastructure

![Sharestreet infra diagram](https://i.imgur.com/1IYTgQv.png)

Sharestreet is a cloud-native application that is highly available and automatically scaled.
It runs on a Kubernetes cluster powered by Google Kubernetes Engine, managed by Terraform.

## Inspiration

We wanted to help food banks and other charitable organizations manage and redistribute their resources. Food insecurity is a major social issue in Canada which has been further exacerbated by the Covid-19 pandemic.

In addition, we also wanted to help hospitals and smaller clinics share PPE and other medical supplies when there was urgent need from the shortages caused by the pandemic.

## What it does

Our application facilitates the sharing of donated supplies between organizations. For example, local food banks may find that they have too much of an item and need to redistribute this item in a prompt manner, whereas others lack items and need a platform in which to put out a call for such items. Very small food banks, pantries, and closets, may not have relationships with larger networks that would allow them to seek out resources. Using our web application, it would be fast and convenient for them to put out a call to multiple food banks at once expressing their need. Or, if a foodbank had too much of an item, they can match their excess supplies better by finding a foodbank that is in need of such item.

## How we built it

ShareStreet is a cloud-native webapp with a React frontend and an Express backend. It runs on Google Kubernetes Engine and is highly available and automatically scaled.

## Challenges we ran into

Provisioning and management of Kubernetes on GKE was challenging as it was our first time interacting with GCP. However, we do have AWS Kubernetes experience, which allowed us to swiftly resolve many issues we encountered.

It was difficult to get the frontend and backend to communicate properly with each other, and the database was also new to us. Fortunately Firestore has a very straightforward library for node.js which allowed us to easily implement APIs that interact with Firestore.

## Accomplishments that we're proud of

We're glad that we were able to make a cloud-native application for easy distribution of resources between charity organizations. We hope many will make use of our application to share resources and minimize resource waste due to product expiry.
## Inspiration

I was doing an application last week when one of the questions about diversity prompted me into thinking about how we take things for more than just granted. It struck me that blind people aren't able to interact with the internet (shop on e-commerce websites, research to learn, play video games, social media - the list is quite endless). I then asked my dad, who works in the computer-science industry, about whether the visually impaired and blind are able to contribute to their likeness in the workforce when I realized how far off the scales were tipped.

![Blind person](https://assileye.com/blog/wp-content/uploads/2021/04/techfortheblind-ft.jpg)

*Assil Eye Institute*

If I were in their shoes, then I would immediately build such a software to be connected like and with everyone else, but in their case I would be blind so I wouldn't be able to, luckily I'm not so I know the right thing to do would be to build a software for them. Also, I realized that for people like me who are quite lazy or would like parts of their browser processes being automated (like telling it to reserve my tennis court from 5-6 PM rather than doing it myself), an AI browser might also sound quite valuable. Rather more of a business market "want" than a "need", I think regardless it's novelty in the market and it's usefulness for a customer segment (did a bit of customer discovery as well reaching out to centers and found out directly from people that they are very much look forward to it) makes this idea worth venturing about this hackathon in CalHacks.

## What it does

It's the same as your regular web browser application (Chrome, Safari, Brave, etc) in the way you surf the web, but in addition, you can do so freely while being blind-folded. Well, how does that work? Also, controlled by text and/or speech input (pressing the space bar for over a second starts a recording and leaving it ends the recording), any user's instructions are followed by the browser. When a user lands on a page, a quick summary of the page is read including the nav bar components, etc so the user can navigate and explore the web just like any other person. A user can ask the browser to click on any element/part of the page and also fill in information on the page via true NLP. They can also open/close tabs, search using the search engine, ask questions (variant of RAG approach taken) over the current page, save the page locally/print the page, and so many other tasks that the current set of agents have to offer. Every time something new is displayed on the browser (ex. going to a different page) they are notified so they can take judgements.

## How we built it

Using the PyQt5 Browser Development framework, I started my code by building a browser. I organized my prompt engineering agentic framework using Fetch AI's agentic system. For prompting questions with images/text I used Gemini's models. I also used Gemini's bounding box model for detecting where to proceed next on the page. This was not too accurate, so I coupled it with my algorithm I wrote where I take the html code of a website and parsed it down to the important segments (removing PII and unnecessary contents) in order to save tokens and decide based on html where to move next as a backup. After much testing, I settled on Groq for the decision-making segments of the LLM chain for its speed. I also used it for the STT part where the user has the option to speak in our application. The TTS part was handled by DeepGram and other voice agent integrations. Building requires testing and I tested this by imitating a blind person by being blind. One such successful testing included making accounts on websites I have never visited before.

## Challenges we ran into

Parsing the HTML took one of the longest parts due to its implicit complexity. Event listeners attached to elements across the DOM tree. Event listeners on images like a hamburger icon which has no text so a mapping is needed. These multiple edge cases had to be considered before reaching the threshold where it was doing perfect on every website as it is doing now. Another challenge I ran into was fine-tuning. The accuracy of this model was at around 60% and it was a hard and enduring work to get it to around 95% where it is currently at right now.

## Accomplishments that we're proud of

Controlling the browser by "having a 2-street conversation" with it simply blows my mind and really changes the way one surfs the web. It's really fun and really useful at the same time.

## What we learned

How to build your own browser. How to fine tune the boundary box model with Gemini through prompt engineering to extract high accurate insights. Learned documentation for multiple voice agent companies that were integrated.

## What's next for BrowseBlind

Perfecting the software model and finding ways to even further reduce token consumption in order to push this into the market as soon as possible. Also, building "background tabs" a feature I didn't have time to finish but essentially you can give a tab a task (ex. find the part on the wikipedia page that talks about fourier transform or find the contact page for X, Y, Z company) and the tab does the task in the background and comes back up when finished.