## Inspiration

1.7 billion people today have cell phones but no Internet connectivity. A large portion of this demographic further faces socioeconomic challenges such as low incomes, illiteracy, and poor hygienic conditions, leaving them vulnerable to unhealthy lifestyles and malady as they lack information. Several of our team members have been in such environments where a lack of awareness and stigmatisation of mental and sexual health have led to depression and suicides, discrimination (particularly against women), and the continuation of poverty cycles. We built IEve to tackle this issue.

## What it does

IEve is a virtual assistant available on phone providing critical information even when users cannot connect to the Internet. A user can call an IEve number and ask a question, particularly for topics of which they lack awareness and/or that may be taboo in their communities. IEve then automatically generates a response to their query using crowd-sourced information that is peer-reviewed and verified by experts. IEve then reads out the answer over the phone so that it is accessible to any type of user.

IEve always ensures high confidence in its answers. If it is not confident, it admits so and automatically opens the question to experts on Einstein. An unanswered question is posted to IEve’s web forums, where users such as doctors, NGOs, and other experts can write responses. Once a response is written, it is peer-reviewed as other experts upvote or downvote it. Once it passes a certain threshold, the response is then sent to selected expert moderators who can then finalise its validity. It is then automatically added to the knowledge base on which IEve’s automated model is trained.

As of now, IEve is specialised in topics on mental and sexual health, which are particularly taboo in countries like India and other parts of the developing world. However, IEve is set up for scalability and will be able to handle any requests as more crowdsourced data is generated.

## How we built it

Our system has multiple components mentioned below:

Voice assistant on phone - We use Twilio to provide a way for users to talk with IEve. Twilio handles incoming calls from the user and, as defined by a Node.js script on an Azure VM server, sends the audio stream to Google Cloud’s cutting-edge speech-to-text model for real-time call transcription. The transcribed query from the user is parsed and forwarded to a Q&A server. The response from the Q&A server is then parsed, and the answer is extracted and read out and/or texted to the user using Twilio. If a user calls and an answer is not available, the question is posted to Einstein through a REST API call. The speech assistance makes IEve accessible to all users, even those who are illiterate or unfamiliar with smartphones. Cell phone integration makes critical information available without Internet access.

Q&A - Our query answering system is built using Azure’s state-of-the-art QnA maker. We generate a knowledge base by scraping verified articles and automatically extracting question answer pairs. The knowledge base is extendable by either directly adding question-answer pairs or by adding more articles to scrape information from. Over time the community efforts on Einstein will organically grow a rich knowledge base.

Einstein - The admin panel is built using the MERN stack (MongoDb, ExpressJS, ReactJS, and NodeJS). We followed a microservice based architecture and deployed our front-end application and backend API using Heroku. The QnA model scrapes the approved page on a daily basis to add to the knowledge base.

## Challenges we ran into

Our main challenge was integrating all our systems so that all of our applications could seamlessly communicate with each other. We used various APIs including Twilio and Node.js frameworks such as WebSockets and MongoDB as intermediaries that handled communication between systems. We were also challenged initially by the non-robustness of our Azure QnA model, which we addressed through question generation and additional data collection.

## Accomplishments that we're proud of

We are quite proud of how we integrated all our systems so that we could leverage the strengths of Azure, Google Cloud, and the Twilio APIs. Our whole application is entirely automatic! Our application is very scalable, and as we hope to help people with information, we are proud that this should increase our scope for impact with time. We are happy that we learned so much in such a short time.

## What we learned

We learned a lot about using Azure, Google Cloud, and Twilio and about web development, NLP, and application design.

## What's next for IEve: An Organically Improving Voice Assistant for All Information for Everyone

Going forward, we will continue to develop IEve to handle more topics more robustly and aim to promote Einstein and begin crowdsourcing with improved automatic knowledge extraction and updates.

## Try it out

Call +1 (205) 579-8843 to see IEve in action
Check out Einstein at: <https://oosharma.github.io/einstein/>
## Inspiration ✨

For many people, playing music serves as a language where they can communicate complex feelings and relive their most cherished memories. It has remained a therapeutic medium for centuries, where people can express their vulnerabilities and seek comfort among others. However, accessing sheet music is challenging for several musicians due to the high cost of music libraries and the difficulty of transcribing music. Thus, we created TuneScriber, a web app that allows users to convert their favourite songs into sheet music! It separates the audio into distinct tracks based on instrumentation, which provides musical education more accessible to musical groups and solo musicians alike.

## What it does 🚀

Simply upload your audio file and watch the magic happen! Our product will split the audio into distinct tracks, enabling you to play each part separately, preview the sheet music, and download it as a PDF file.

## How we built it 🔧

* We used **React**, **TailwindCSS**, and **Framer Motion** to create a scalable project that provides an interactive and responsive experience for everyone!
* **Flask** was used for server-side integration by sending the audio file to the backend for track separation, audio-to-MIDI conversion and sheet music generation.
* We used **Music.ai** to split the song into individual tracks based on instrumentation, including vocal, bass, string, and more!
* Next, the generated track files are fed into **Basic Pitch** for MIDI conversion, which enables easy editing and integration with other music production software (i.e. MuseScore).
* Finally, these MIDI files were converted into sheet music using **Mido**, which provides musicians with a visual representation of music with traditional notation. We also used **Whisper.ai** for reliable lyric generation, where arrays of verses are routed to the frontend so musicians can enjoy their favourite songs.

## Challenges we ran into 🥲

* Lots of typos! While they are silly mistakes, we spent hours trying to fix bugs that ended up being spelling and grammatical errors.
* Whether it's CORS, merge conflict, or API integration, we ran into several challenges when integrating our front-end and back-end. A big contributor was the system differences between MacOS and Windows, where Pip package, directory, and dependency errors extended our hacking process by 10+ hours :(

## Accomplishments that we're proud of 🌟

* Successfully creating a responsive and intuitive UI to enhance the user experience by providing scroll features, engaging transitions, and drag-and-drop functionality
* Integrating uploaded files onto our backend server for audio processing, which enables our product to convert audio files into MIDI and download them as sheet music PDFs
* Despite how frustrated we were, we stayed resilient throughout our Hack the 6ix journey and overcame every obstacle we encountered this weekend!

## What we learned 🍎

* How to create scalable React components to simplify the front-end process! We also explored various UI libraries like **Aceternity UI** to create smooth, responsive elements for enhanced user experience (i.e. moving border buttons).
* How to resolve merge conflicts and collaborate effectively using **Git**. Each team member created their branch to work on their tasks, and we merged these changes to ensure punctual project completion.
* The integration of several packages, dependencies and APIS, including **Basic Pitch** and **Whisper.ai**. for server-side routing. It allowed our product to create individual MIDI files and deliver lyric transcriptions for enhanced musical experience.

## What's next for TuneScriber 👀

* Adding an in-app MIDI editor to enable user personalization! Whether it’s changing the key, the tempo, or the difficulty, they should have the opportunity to tailor results to fit their needs!
* Since the musical timbre of drum sets functions differently compared to other instrument counterparts, optimizing our product to generate percussion sheet music can greatly improve user experience and make music more accessible for large ensembles.
* Utilize database integration to store previous conversions so users can access their sheet music whenever and wherever!
* Adding lyrics to the sheet music (rather than as a separate file/pop-up) so musicians can play their instruments and sing their hearts out 🎤
* Improve conversion efficiency by reducing the number of API calls to turn our multi-step process into one-step
## Inspiration

Partially inspired by the Smart Cities track, we wanted our app to have the direct utility of ordering food, while still being fun to interact with. We aimed to combine convenience with entertainment, making the experience more enjoyable than your typical drive-through order.

## What it does

You interact using only your voice. The app automatically detects when you start and stop talking, uses AI to transcribe what you say, figures out the food items (with modifications) you want to order, and adds them to your current order. It even handles details like size and flavor preferences. The AI then generates text-to-speech audio, which is played back to confirm your order in a humorous, engaging way. There is absolutely zero set-up or management necessary, as the program will completely ignore all background noises and conversation. Even then, it will still take your order with staggering precision.

## How we built it

The frontend of the app is built with React and TypeScript, while the backend uses Flask and Python. We containerized the app using Docker and deployed it using Defang. The design of the menu is also done in Canva with a dash of Harvard colors.

## Challenges we ran into

One major challenge was getting the different parts of the app—frontend, backend, and AI—to communicate effectively. From media file conversions to AI prompt engineering, we worked through each of the problems together. We struggled particularly with maintaining smooth communication once the app was deployed. Additionally, fine-tuning the AI to accurately extract order information from voice inputs while keeping the interaction natural was a big hurdle.

## Accomplishments that we're proud of

We're proud of building a fully functioning product that successfully integrates all the features we envisioned. We also managed to deploy the app, which was a huge achievement given the complexity of the project. Completing our initial feature set within the hackathon timeframe was a key success for us. Trying to work with Python data type was difficult to manage, and we were proud to navigate around that. We are also extremely proud to meet a bunch of new people and tackle new challenges that we were not previously comfortable with.

## What we learned

We honed our skills in React, TypeScript, Flask, and Python, especially in how to make these technologies work together. We also learned how to containerize and deploy applications using Docker and Docker Compose, as well as how to use Defang for cloud deployment.

## What's next for Harvard Burger

Moving forward, we want to add a business-facing interface, where restaurant staff would be able to view and fulfill customer orders. There will also be individual kiosk devices to handle order inputs. These features would allow *Harvard Burger* to move from a demo to a fully functional app that restaurants could actually use. Lastly, we can sell the product by designing marketing strategies for fast food chains.