## Inspiration

We started off having the idea of how people could progressively integrate technology into their lives, and with all the smart electronic device and household appliances, we want to further integrate technology into the textile industry. Our team is particularly interested in computer vision and image processing so we want to create a jacket (cuz it is winter coming) that can be applied to security and healthcare.

## What it does

1. Recent reports from PoliceOne.com says that it is vital to integrate these body-worn cameras into their duty to increase efficiency and effectiveness, so we are planning to build these cameras into officers' and police-dogs' jackets for early detection of targeted individuals from police records databases, especially during civil unrest.
2. Our device would provide a platform for the visually-impaired to get real-time information of objects, signs, and other people in front of them. This program can be connected to smart blind glasses with headphones to benefit the disabled population.

## How I built it

We used a Raspberry Pi camera as a reasonably-sized device to fit in the jacket, then we implemented openCV to capture real-time data with its built-in object recognition functions.

## Challenges I ran into

First, two out of three people in our team are better with software so we needed to spend some time getting used to the hardware tools. Then comes the software problem, we initially started off with openCV and we tried different tutorials to install it into the Raspberry Pi. That took us a huge amount of time and we did not get the code to compile.

Then we were thinking of implementing Google Vision API instead and spend quite a while to realize that the API does not take in video inputs but static images. Therefore, we decided to go back to OpenCV, tried out using pre-compiled binary openCV, but we ran into some error. Finally, we realized that we need to resize our OS's partition to have enough space on our SD card to run OpenCV.

## Accomplishments that I'm proud of

Our team got to explore the potential applications of several hardware tools, learned some cool problem-solving skills, especially when we stepped through how to get OpenCV installed. We also finally came up with an idea that can benefit public goods in terms of security and health safety.

## What's next for Konnex Jacket

The next steps are we need to find a portable battery to use as a power source for our cameras. We are also trying to put in some fluorescent reflecting paint as a safety feature. To serve as a police public safety device, we need to connect the camera with the police records database, and to serve as a health assistance tool, we need to filter the information reported to the user.
## Inspiration

Sometimes in lecture you need to point to something tiny on the presentation but no one really knows what you're pointing to. So we decided to build something that can read where you are pointing using a camera, and points a laser in that direction, which makes engagement in lectures and presentations much more accessible. We also realized that this idea actually branches off into a lot of other potential accessibility applications; it allows robotic control with pure human actions as input. For example, it could help artists paint large canvases by painting at where they point, or even be used as a new type of remote control if we replaced the laser with an RF signal.

## What it does

It tracks where the user's forearm is using a fully custom-built computer vision object detection program. All dimensions of the forearm are approximated via only one camera and it is able to generate a set of projected XY coordinates on the supposed presentation for the laser to point to. This corresponds to where the user is pointing at.

## How we built it

We heavily used OpenCV in Python to built the entire computer vision framework, which was tied to a USB webcam. Generated projection points were sent to an ESP32 via wifi, which fed separate coordinates to a dual-servo motor system which then moves the laser pointer to the correct spot. This was done using arduino.

## Challenges we ran into

First of all, none of us had actually used OpenCV on a project this size, especially not for object tracking. This took a lot of learning on the spot, online tutorials and experimenting
There were also plenty of challenges that all revolved around the robustness of the system. Sometimes the contour detection software would detect multiple contours, so a challenge was finding a way to join them so the system wouldn't break. The projection system was quite off at the start, so a lot of manual tuning had to be done to fix that. The wifi data transmission also took a long time to figure out, as none of us had ever touched that stuff before.

## Accomplishments that we're proud of

We're quite proud of the fact that we were able to build a fully functional object tracking system without any premade online code in such a short amount of time, and how robust it was in action. It was also quite cool to see the motors react in real time to user input.

## What we learned

We learned some pretty advanced image processing and video capture techniques in OpenCV, and how to use the ESP32 controller to do stuff.

## What's next for Laser Larry

The biggest step is to make the projection system more accurate, as this would take a lot more tuning. Another camera also wouldn't hurt to get more accurate readings, and it would be cool to expand the idea to more accessibility applications discussed above.
## Inspiration

Our project, "**Jarvis**," was born out of a deep-seated desire to empower individuals with visual impairments by providing them with a groundbreaking tool for comprehending and navigating their surroundings. Our aspiration was to bridge the accessibility gap and ensure that blind individuals can fully grasp their environment. By providing the visually impaired community access to **auditory descriptions** of their surroundings, a **personal assistant**, and an understanding of **non-verbal cues**, we have built the world's most advanced tool for the visually impaired community.

## What it does

"**Jarvis**" is a revolutionary technology that boasts a multifaceted array of functionalities. It not only perceives and identifies elements in the blind person's surroundings but also offers **auditory descriptions**, effectively narrating the environmental aspects they encounter. We utilize a **speech-to-text** and **text-to-speech model** similar to **Siri** / **Alexa**, enabling ease of access. Moreover, our model possesses the remarkable capability to recognize and interpret the **facial expressions** of individuals who stand in close proximity to the blind person, providing them with invaluable social cues. Furthermore, users can ask questions that may require critical reasoning, such as what to order from a menu or navigating complex public-transport-maps. Our system is extended to the **Amazfit**, enabling users to get a description of their surroundings or identify the people around them with a single press.

## How we built it

The development of "**Jarvis**" was a meticulous and collaborative endeavor that involved a comprehensive array of cutting-edge technologies and methodologies. Our team harnessed state-of-the-art **machine learning frameworks** and sophisticated **computer vision techniques** to get analysis about the environment, like , **Hume**, **LlaVa**, **OpenCV**, a sophisticated computer vision techniques to get analysis about the environment, and used **next.js** to create our frontend which was established with the **ZeppOS** using **Amazfit smartwatch**.

## Challenges we ran into

Throughout the development process, we encountered a host of formidable challenges. These obstacles included the intricacies of training a model to recognize and interpret a diverse range of environmental elements and human expressions. We also had to grapple with the intricacies of optimizing the model for real-time usage on the **Zepp smartwatch** and get through the **vibrations** get enabled according to the **Hume** emotional analysis model, we faced issues while integrating **OCR (Optical Character Recognition)** capabilities with the **text-to speech** model. However, our team's relentless commitment and problem-solving skills enabled us to surmount these challenges.

## Accomplishments that we're proud of

Our proudest achievements in the course of this project encompass several remarkable milestones. These include the successful development of "**Jarvis**" a model that can audibly describe complex environments to blind individuals, thus enhancing their **situational awareness**. Furthermore, our model's ability to discern and interpret **human facial expressions** stands as a noteworthy accomplishment.

## What we learned

# Hume

**Hume** is instrumental for our project's **emotion-analysis**. This information is then translated into **audio descriptions** and the **vibrations** onto **Amazfit smartwatch**, providing users with valuable insights about their surroundings. By capturing facial expressions and analyzing them, our system can provide feedback on the **emotions** displayed by individuals in the user's vicinity. This feature is particularly beneficial in social interactions, as it aids users in understanding **non-verbal cues**.

# Zepp

Our project involved a deep dive into the capabilities of **ZeppOS**, and we successfully integrated the **Amazfit smartwatch** into our web application. This integration is not just a technical achievement; it has far-reaching implications for the visually impaired. With this technology, we've created a user-friendly application that provides an in-depth understanding of the user's surroundings, significantly enhancing their daily experiences. By using the **vibrations**, the visually impaired are notified of their actions. Furthermore, the intensity of the vibration is proportional to the intensity of the emotion measured through **Hume**.

# Ziiliz

We used **Zilliz** to host **Milvus** online, and stored a dataset of images and their vector embeddings. Each image was classified as a person; hence, we were able to build an **identity-classification** tool using **Zilliz's** reverse-image-search tool. We further set a minimum threshold below which people's identities were not recognized, i.e. their data was not in **Zilliz**. We estimate the accuracy of this model to be around **95%**.

# Github

We acquired a comprehensive understanding of the capabilities of version control using **Git** and established an organization. Within this organization, we allocated specific tasks labeled as "**TODO**" to each team member. **Git** was effectively employed to facilitate team discussions, workflows, and identify issues within each other's contributions.

The overall development of "**Jarvis**" has been a rich learning experience for our team. We have acquired a deep understanding of cutting-edge **machine learning**, **computer vision**, and **speech synthesis** techniques. Moreover, we have gained invaluable insights into the complexities of real-world application, particularly when adapting technology for wearable devices. This project has not only broadened our technical knowledge but has also instilled in us a profound sense of empathy and a commitment to enhancing the lives of visually impaired individuals.

## What's next for Jarvis

The future holds exciting prospects for "**Jarvis.**" We envision continuous development and refinement of our model, with a focus on expanding its capabilities to provide even more comprehensive **environmental descriptions**. In the pipeline are plans to extend its compatibility to a wider range of **wearable devices**, ensuring its accessibility to a broader audience. Additionally, we are exploring opportunities for collaboration with organizations dedicated to the betterment of **accessibility technology**. The journey ahead involves further advancements in **assistive technology** and greater empowerment for individuals with visual impairments.