## Inspiration

it's really fucking cool that big LLMs (ChatGPT) are able to figure out on their own how to use various tools to accomplish tasks.

for example, see Toolformer: Language Models Can Teach Themselves to Use Tools (<https://arxiv.org/abs/2302.04761>)

this enables a new paradigm self-assembling software: machines controlling machines.

what if we could harness this to make our own lives better -- a lil LLM that works for you?

## What it does

i made an AI assistant (SMS) using GPT-3 that's able to access various online services (calendar, email, google maps) to do things on your behalf.

it's just like talking to your friend and asking them to help you out.

## How we built it

a lot of prompt engineering + few shot prompting.

## What's next for jarbls

shopping, logistics, research, etc -- possibilities are endless

* more integrations !!!
the capabilities explode exponentially with the number of integrations added
* long term memory

come by and i can give you a demo
## Inspiration

One of our teammate’s grandfathers suffers from diabetic retinopathy, which causes severe vision loss.
Looking on a broader scale, over 2.2 billion people suffer from near or distant vision impairment worldwide. After examining the issue closer, it can be confirmed that the issue disproportionately affects people over the age of 50 years old. We wanted to create a solution that would help them navigate the complex world independently.

## What it does

### Object Identification:

Utilizes advanced computer vision to identify and describe objects in the user's surroundings, providing real-time audio feedback.

### Facial Recognition:

It employs machine learning for facial recognition, enabling users to recognize and remember familiar faces, and fostering a deeper connection with their environment.

### Interactive Question Answering:

Acts as an on-demand information resource, allowing users to ask questions and receive accurate answers, covering a wide range of topics.

### Voice Commands:

Features a user-friendly voice command system accessible to all, facilitating seamless interaction with the AI assistant: Sierra.

## How we built it

* Python
* OpenCV
* GCP & Firebase
* Google Maps API, Google Pyttsx3, Google’s VERTEX AI Toolkit (removed later due to inefficiency)

## Challenges we ran into

* Slow response times with Google Products, resulting in some replacements of services (e.g. Pyttsx3 was replaced by a faster, offline nlp model from Vosk)
* Due to the hardware capabilities of our low-end laptops, there is some amount of lag and slowness in the software with average response times of 7-8 seconds.
* Due to strict security measures and product design, we faced a lack of flexibility in working with the Maps API. After working together with each other and viewing some tutorials, we learned how to integrate Google Maps into the dashboard

## Accomplishments that we're proud of

We are proud that by the end of the hacking period, we had a working prototype and software. Both of these factors were able to integrate properly. The AI assistant, Sierra, can accurately recognize faces as well as detect settings in the real world. Although there were challenges along the way, the immense effort we put in paid off.

## What we learned

* How to work with a variety of Google Cloud-based tools and how to overcome potential challenges they pose to beginner users.
* How to connect a smartphone to a laptop with a remote connection to create more opportunities for practical designs and demonstrations.
How to connect a smartphone to a laptop with a remote connection to create more opportunities for practical designs and demonstrations.
How to create docker containers to deploy google cloud-based flask applications to host our dashboard.
How to develop Firebase Cloud Functions to implement cron jobs. We tried to developed a cron job that would send alerts to the user.

## What's next for Saight

### Optimizing the Response Time

Currently, the hardware limitations of our computers create a large delay in the assistant's response times. By improving the efficiency of the models used, we can improve the user experience in fast-paced environments.

### Testing Various Materials for the Mount

The physical prototype of the mount was mainly a proof-of-concept for the idea. In the future, we can conduct research and testing on various materials to find out which ones are most preferred by users. Factors such as density, cost and durability will all play a role in this decision.
## Inspiration

As a team with diverse skill sets, ranging from computer science to art and philosophy, we wanted to build something that encapsulated a problem and an interest that we all shared. This intersection was design. We realized, we've all gravitated towards design at some point in our academic careers. However, we realized that most of our design experience was related to drawing or ideating about products, and not so much of actual implementation in tools like Figma.

We realized that this is because there is an inherent barrier to learning how to use tools like Figma before designs and ideas can be realized. On this note, last year, almost 18% of surveyed marketers revealed that they spend an average of over 20 hours per week creating visual content. Inspired by the mass gains in productivity accomplished through tools such as ChatGPT for programmers (i.e. no more forgetting how to implement Union-Find then searching up on Google how to implement it (๑•̀ㅂ•́)و) , our team was inspired to create an AI-driven vision/language design assistant that can also decrease the amount of time new designers spend on the "in-between" processes, such as transferring their drawings over to a Figma board, or creating a component that is styled in a certain way.

Thus, our mission is to empower people **new to UI/UX graphic design** to spend more time thinking creatively about an idea and less time figuring out how to get a curve at the perfect angle on Figma.

## What it does

Our tool is an interactive AI-powered design assistant manifested as a Figma plugin, that streamlines and personalizes your design process by converting a drawing to a Figma design in seconds. Once the design is on the screen, you can then (in Figma!) use ChatGPT to ask for components that fit the style *you* are going for. This allows you to not waste time trying to re-draw rectangles, make sure the borders are rounded correctly, check the color... (you get the point). We wanted to leverage the power of large language models to interpret this certain styles of designs we are thinking of automatically, and give to us in Figma, thereby minimizing potentially redundant work. Since it is a plugin, there is minimal barrier to entry for new designers trying to get engaged in the visual design space.

We note for full transparency that the description above is the ideal product. While we definitely made strides and had major accomplishments our team was proud of, our final coded product was a portion of this (more specifically, two components of the platform) that we didn't have time to fully integrate. However, we did make our ideas of turning some form of HTML to Figma Components a reality. We elaborate more on this below.

## How we built it

Our team decided to split the project up into two primary components:
1) Going from drawing --> HTML
2) Creating a Figma plugin that can go from HTML --> Figma elements.

The first part was developed by iterating on Microsoft's SketchToCode. While this was a useful starting point, most of the code was outdated to some degree, so our team had to refactor the code base to get the model to train on a set of hand-labeled drawing datasets. The model we trained was an instance segmentation model hosted in Azure to determine common visual components in drawings such as Buttons, TextFields, Text, etc.

The second part involved using Figma's Plugin API which uses TypeScript and HTML, and drawing upon APIs such as OpenAI's ChatGPT as well as other Figma plugins that have functionality to translate websites from URLs into Figma components. While we were developing, we were able to track our progress by using Figma Desktop to visualize the UI/UX of our plugin, and observing how it translated to the output that we desired. We also hosted a server in Node.js and Express that makes calls to the ChatGPT API.

## Challenges we ran into

During the hackathon, our team ran into several bugs with existing APIs. For instance, the team members working on the drawing to HTML portion of our project tried to iterate on Microsoft's SketchToCode AI, which is reasonably outdated. Thus, a lot of time was spent trying to refactor some of the code that we initially hoped would work out of the box. Another example is the incorporation ChatGPT into to the Figma plugin. Our team members were new to working with TypeScript, so there were several errors/debugging incorporating some node modules with the Figma plugins.

## Accomplishments that we're proud of

We're proud after struggling for a long time on trying to incorporate ChatGPT into a Figma plugin (mostly due to Typescript issues), we were able to finally get to a point where users could input data into ChatGPT through a Figma plugin, and achieve some visual output on Figma components. We're also proud that we were able to train an instance segmentation model that obtained ~84% accuracy on predicting common UI components in drawn designs.

Most importantly, we're proud that our team was able to stumble across an idea that we all thought was cool, and that we could see ourselves carrying out to completion over the next few weeks.

## What we learned

From the technical standpoint, we learned that while it is usually a good idea to use existing APIs to get a project off the ground, sometimes its OK to start something new from scratch. We feel that a lot of the friction that prevented us from getting our product off the ground was attempting to use APIs that were inherently flawed / buggy.

We also found a lot of interesting ways to interact with Figma programmatically . Prior to this hackathon, none of our team members had built Figma plugins.

## What's next for FigmAI

Next steps include a full integration of the drawing to code pipeline with the html to Figma component pipeline. (i.e. adding images through Figma, fixing issues with styling, etc.)

Moving forward, our team was also thinking about a similar pipeline but for CAD (which has an even higher barrier to entry for most). In other words, what if you could draw a component on paper, and then port that design to a CAD model to start with? Our team brainstormed ways to incorporate Reinforcement Learning algorithms to teach an agent in CAD / Figma to perform a set of actions to minimize the difference between a ground truth output and a model output.