## CodeBoard: Run Your Handwritten Code!

![CBicon](https://raw.githubusercontent.com/ayung8/CodeBoard/master/CBicon_s.png)

Team Members: Adrian Yung, Jessie Muir, Emily Wang, Vincent Borja

[Github](https://github.com/ayung8/CodeBoard)

**Summary**

CodeBoard is a mobile friendly web application built with Python that leverages the latest in AI and Online Automation techniques. CodeBoard allows users to run code wherever it can be found – on paper, on a whiteboard, or even scribbled in sand – simply by taking a picture.

CodeBoard is hosted on Google’s App Engine and utilizes Google’s Cloud Computing as well as Sphere Engine’s Automation Services. The ability to easily parse text from images using Google’s Vision AI paired with Sphere Engine’s online compiler all contributes towards the goal of allowing a seamless conversion from written code to output.

The application can be used by many: students taking a picture of code scribbled in class by the professor, interviewers quickly running code that is written on the blackboard, and professors checking to see if the solution to an exam question containing 30 for-loops actually works. Evidently, there are a wide range of applications where it would be advantageous to have a quick and easy way to run code simply by taking a picture.

![CBFlowchart](https://raw.githubusercontent.com/ayung8/CodeBoard/master/CBflowcharticon.png)

**Technical Information**

CodeBoard leverages 2 main APIs: Google Vision and Sphere Engine. Google’s Vision API is fed pictures to analyze and parse text. With this text, CodeBoard extracts relevant information to rebuild bits and pieces of code that might have been lost in translation. Function names, return types, and arguments are all reconstructed to successfully run the code. In addition, CodeBoard validates the code before finally passing it to Sphere Engine’s Online Compiler API for execution. The utilization of both Google’s Vision API and Sphere Engine’s API streamlines the process and drastically reduces the time needed to translate code from paper to output. CodeBoard is hosted using Google’s App Engine.

**Features & How to Use**

Take a picture of some code! Select the appropriate programming language, as well as indicate whether it is a function or to run it as is (a whole program). You are able to make edits before the code is run.
Currently, CodeBoard supports code written in Javascript and C. Plans to expand to Python, Java, and C++ are already underway!

**Examples**

![Example2](https://github.com/ayung8/CodeBoard/blob/master/example1_res.jpg?raw=true)
![Example1](https://github.com/ayung8/CodeBoard/blob/master/example2_res.jpg?raw=true)
![EgBeginScreen](https://github.com/ayung8/CodeBoard/blob/master/beginscreen_res.png?raw=true)
![EgOutput](https://github.com/ayung8/CodeBoard/blob/master/exampleanswer_res.png?raw=true)
## 💡Inspiration

Sometimes it's hard to sit in front of a screen all day and write code. It definitely takes a **toll on your health**. So why not go *old-school* and write the code on a pen and paper so that it doesn't affect your Health.

Also this can be helpful for the students who are getting on with coding and have *lack of resources* (like laptops). So they have to use their mobile phone to write and compile their code **which takes a lot of time and effort**.

## 💻What it does

**Codifier** is a web based code converter which can translate your *handwritten code* into *actual typed code* using **OCR** in just a few seconds. Don't worry about typing all the code, just scan and upload and *Et-voilà!* It also Compiles the code in various languages like **C,C++,Java,Python**.

## 🔷Steps to Use

1. Click on get Started.
2. Select the option to either click a photo or choose from gallery.
3. Upload your Scanned code and click on **Convert**.
	* This process will take you to the compiler.
4. Select your programming language.
5. After getting your code on the screen, you can edit the code if you want.
6. Click the Compile button to compile and run it into the output screen.

## 🔨How we built it

* React Js: For Frontend
* Tailwind CSS: For styling
* Figma: For Designing. (You can see it [here](https://www.figma.com/proto/geIJeadw8u9XAQ4BrdAMp2/Pictocode?node-id=2%3A2&starting-point-node-id=2%3A2))
* Node Js: For Backend Server
* API Google Cloud Vision and [JDoodle API](https://docs.jdoodle.com/compiler-api/compiler-api)

## 💻AI Code Recognition Challenges

Codifier uses Google Cloud Vision API for recognizing the handritten code and converting into typed code, It also uses the JDoodle API to compile and run the code into different programming languages like **C,C++,Java,Python**.

## 🧠Use of Google Cloud Vision API

* Handritten code Recognition.
* Conversion into Type code.

## 🏅Challenges We ran into

* Communication gap due to Online Hackathons.
* The Biggest Challenge we ran into was to configure the API to recognize and convert the code.
* Compiling and running the code into different languages.

## 📖What we learnt

1. Team Building
2. Working with:
3. React Js
4. Figma
5. Multer
6. Git and Github
7. APIs
8. PWA (Progressive Web Applications)

## 🚀What's Next for Codifier

* Adding the Camera feature for recognizing the code as a built-in feature.
* Authenticating the user and storing their code in the Database.
* Visualizing the code.
* Adding input option in code compilation for user.
## What Does "Catiator" Mean?

**Cat·i·a·tor** (*noun*): Cat + Gladiator! In other words, a cat wearing a gladiator helmet 🐱

## What It Does

*Catiator* is an educational VR game that lets players battle gladiator cats by learning and practicing American Sign Language. Using finger tracking, players gesture corresponding letters on the kittens to fight them. In order to survive waves of fierce but cuddly warriors, players need to leverage quick memory recall. If too many catiators reach the player, it's game over (and way too hard to focus with so many chonky cats around)!

## Inspiration

There are approximately 36 million hard of hearing and deaf individuals live in the United States, and many of them use American Sign Language (ASL). By learning ASL, you'd be able to communicate with 17% more of the US population. For each person who is hard of hearing or deaf, there are many loved ones who hope to have the means to communicate effectively with them.

### *"Signs are to eyes as words are to ears."*

As avid typing game enthusiasts who have greatly improved typing speeds ([TypeRacer](https://play.typeracer.com/), [Typing of the Dead](https://store.steampowered.com/agecheck/app/246580/)), we wondered if we could create a similar game to improve the level of understanding of common ASL terms by the general populace. Through our Roman Vaporwave cat-gladiator-themed game, we hope to instill a low barrier and fun alternative to learning American Sign Language.

## Features

**1. Multi-mode gameplay.**
Learn the ASL alphabet in bite sized Duolingo-style lessons before moving on to "play mode" to play the game! Our in-app training allows you to reinforce your learning, and practice your newly-learned skills.

**2. Customized and more intuitive learning.**
Using the debug mode, users can define their own signs in Catiator to practice and quiz on. Like Quizlet flash cards, creating your own gestures allows you to customize your learning within the game. In addition to this, being able to see a 3D model of the sign you're trying to learn gives you a much better picture on how to replicate it compared to a 2D image of the sign.

## How We Built It

* **VR**: Oculus Quest, Unity3D, C#
* **3D Modeling & Animation**: Autodesk Maya, Adobe Photoshop, Unity3D
* **UX & UI**: Figma, Unity2D, Unity3D
* **Graphic Design**: Adobe Photoshop, Procreate

## Challenges We Ran Into

**1. Limitations in gesture recognition.** Similar gestures that involve crossing fingers (ASL letters M vs. N) were limited by Oculus' finger tracking system in differential recognition. Accuracy in finger tracking will continue to improve, and we're excited to see the capabilities that could bring to our game.

**2. Differences in hardware.** Three out of four of our team members either own a PC with a graphics card or an Oculus headset. Since both are necessary to debug live in Unity, the differences in hardware made it difficult for us to initially get set up by downloading the necessary packages and get our software versions in sync.

**3. Lack of face tracking.** ASL requires signers to make facial expressions while signing which we unfortunately cannot track with current hardware. The Tobii headset, as well as Valve's next VR headset both plan to include eye tracking so with the increased focus on facial tracking in future VR headsets we would better be able to judge signs from users.

## Accomplishments We're Proud Of

We're very proud of successfully integrating multiple artistic visions into one project. From Ryan's idea of including chonky cats to Mitchell's idea of a learning game to Nancy's vaporwave aesthetics to Jieying's concept art, we're so proud to see our game come together both aesthetically and conceptually. Also super proud of all the ASL we learned as a team in order to survive in *Catiator*, and for being a proud member of OhYay's table1.

## What We Learned

Each member of the team utilized challenging technology, and as a result learned a lot about Unity during the last 36 hours! We learned how to design, train and test a hand recognition system in Unity and build 3D models and UI elements in VR.

This project really helped us have a better understanding of many of the capabilities within Oculus, and in utilizing hand tracking to interpret gestures to use in an educational setting. We learned so much through this project and from each other, and had a really great time working as a team!

## Next Steps

* Create more lessons for users
* Fix keyboard issues so users can define gestures without debug/using the editor
* Multi-hand gesture support
* Additional mini games for users to practice ASL

## Install Instructions

To download, use password "Treehacks" on <https://trisol.itch.io/catiators>, because this is an Oculus Quest application you must sideload the APK using Sidequest or the Oculus Developer App.

## Project Credits/Citations

* Thinker Statue model: [Source](https://poly.google.com/u/1/view/fEyCnpGMZrt)
* ASL Facts: [ASL Benefits of Communication](https://smackhappy.com/2020/04/asl-benefits-communication/)
* Music: [Cassette Tape by Blue Moon](https://youtu.be/9lO_31BP7xY) |
 [RESPECOGNIZE by Diamond Ortiz](https://www.youtube.com/watch?v=3lnEIXrmxNw) |
 [Spirit of Fire by Jesse Gallagher](https://www.youtube.com/watch?v=rDtZwdYmZpo) |
* SFX: [Jingle Lose](https://freesound.org/people/LittleRobotSoundFactory/sounds/270334/) |
 [Tada2 by jobro](https://freesound.org/people/jobro/sounds/60444/) |
 [Correct by Eponn](https://freesound.org/people/Eponn/sounds/421002/) |
 [Cat Screaming by InspectorJ](https://freesound.org/people/InspectorJ/sounds/415209/) |
 [Cat2 by Noise Collector](https://freesound.org/people/NoiseCollector/sounds/4914/)