## Inspiration

Snap! Is like your iPhone live photos (where the picture shows movement before and after slightly) but 100x more memorable. **If you’ve ever taken a photo of a fun event like a concert, kickback, or house party but felt like it couldn’t exactly capture the “vibe”, this is probably for you.** By turning 2d photos into 3d experiences with pretty low latency (~10 seconds automated), it feels like you’re in the room again. **Good times == good memories, so why not start preserving the vibes in between :)**

## What it does

**Snap! applies depth mapping with computer vision to your photos to artificially create a 3D immersive scene.** Then, that photo is rendered with ThreeJS in a 3D environment, so you can drag around in 180 degree angles and see what’s going on as if you were reliving that moment. This prototype is completely functional.

## How we built it

**1) Deciding on the photo input**

This prototype presented a ton of unexpected, interesting technical challenges around cameras, optics, depth estimation, and 3d software like ThreeJS. Simulating a 3D camera with computer vision (depth estimation) and handling a completely different data type than ThreeJS typically uses for displacement maps was more hacky and required more workarounds.

I had to first figure out how to use an iPhone to mimic a 3D camera (such as Reto 3D that takes pictures at various angles and then combines those pictures into a “moving GIF”) as well as stereo cameras, which is known for mimicking how a human perceives the world (with 2 cameras instead of 2 eyes). Traditional methods for stereo cameras work by taking two or more images and estimating a 3D model of the scene. This is done by finding matching pixels in the images and converting their 2D positions into 3D depths. But this traditional method requires special lenses with expensive equipment.

Using an iPhone camera lens to have the same 3D effect is challenging, because it’s purely a 2d image. However, the data format I decided on was a .mov GIF created using the “Bounce effect” for Live Photos. It captures ~1-2 seconds before and after a picture was taken. It’s also the most functional for most, since handling stereo and 3d cameras along with the development process after could be complex.

Afterwards, I used a CLI tool, ffmpeg, to individually extract every frame (around 10-20 frames) for the .mov file.

**2) Depth Estimation on the photos**

Depth estimation is the technical meat of this hack as it provides the immersive effect. It quantifies spatial relationships within pixels, enabling the reconstruction of images into detailed 3D models by interpreting and mapping depth cues.

I ended up applying off-the-shelf model weights from the paper Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data. Foundation Model for Monocular Depth Estimation. I decided on using this model due to it achieving high benchmark results, and being trained on 6M+ unlabelled and labeled images. PyTorch and OpenCV were used for applying the model’s weighs onto my images folder (pure inference, no fine-tuning). The output was a concatenated image of the raw image and its depth map transformed into an image data format. The depth map is a 2D array 564x564 that has values ranging from 0-255 indicating the closeness of the pixel to the camera.

**3) Layering the Depth Map onto the 2D Image in ThreeJS**

This was the tricky part where I had to come up with a work-around solution since ThreeJS doesn’t typically accept depth maps as displacement maps, so it required a bit of learning about 3D in ThreeJS. In Scene.JS, I essentially loaded the 2D image (½ of the inputted concatenated image) as a Canvas Texture and the depth map as a Canvas Texture as well (2/2 of the inputted concatenated image). Then, I loaded each corresponding depth map as a displacement map using ThreejS’s MeshStandardMaterial. As for the plane that the whole 3D environment was living on, a 10 x 10 x 512 x 512 plane was created with the mesh including the plane and the material (MeshStandardMaterial).

OrbitControls from ThreeJS were also loaded to support camera movement (that way, you can explore the image with your mouse as if you were in the scene). For aesthetics, I loaded ambient light settings.

**4) Animating the frames**

By this time, I had successfully outputted a singular 3d photo in ThreeJS. However, I wanted to see if I could make a simple animation loop that would apply depth maps onto its corresponding photo and render it quickly enough so it would look like it was a moving scene. This was completed by creating a custom animation function that would calculate the elapsed time and then determine when to re-render the next frame. Eventually, I landed on specific frame settings that mimicked a GIF, as the photos were moving quickly enough to simulate movement.

## Challenges we ran into

Originally, I applied the depth map as a separate layer from the image canvas texture. This was a bit tricky to troubleshoot because there was essentially zero output from the rendered ThreejS environment. Closely reading the ThreeJS docs helped, and eventually I found the custom function THREE.MeshStandardMaterial that led to the displacement map displaying.

Figuring out whether I would need some sort of calibration for the moving GIF portion was also a challenge. I originally thought of a more complex solution: individually aligning similar points of the same edges across multiple frames in order to account for images where there was more movement occurring. I was worried that the depth maps would be misaligned across various frames. However, aligning the points was not needed since the frames were taken granularly enough that there didn’t seem to be too much diversion from the pin point.

## Accomplishments that we're proud of

Rendering the GIF onto ThreeJS! I originally thought it was only feasible to render an image in the environment, but through a workaround solution using a custom animation function, I was able to render 10-20 frames fast enough to achieve the “movement” effect. Lots of workarounds in this prototype (previous depth map as a displacement map mentioned earlier).

## What we learned

Cameras are difficult to simulate on software! ThreeJS geometry, meshes, and materials were a bit abstract to grasp at first, as it was hard to figure out what would be the equivalent formats for this purpose. Eventually, I decided to have the 2d Image be a texture and then a material, and then layering the depth map as a displacement map through that.

## What's next for Snap!

NerFS instead of depth estimation for turning 2D photos into accurate 3D scenes! It’d be cool to compare this classical CV depth estimation technique with a more recent finding. However, NeRFs take a lot of compute as it needs to perform inference on every x,y,z point. It’d be great to have an NVIDIA RTX 4080 GPU to do that ;)
## Inspiration

Virtual reality is a blooming technology. It has a bright future regardless of which sector we may use it for. From medical training (holographic human body analysis) to entertainment. Particularly for me, its more of an accessory that I believe one day our civilization may be obsessed upon like the television and internet.

## What it does

This app is capable of closing the border between the real and “supposed to be” scientific fantasy world. This app uses the concept of both Augmented reality along with Virtual Reality bringing use one step closer to a different kind of holograms. This app intact is much more superior at what it does than the $4000 Hololens. While Hololens thrives on its projector in the side of the lens and with a very narrow field of view. This app opens up the world of virtual and augmented world in a way which makes it way cheaper alternative than its counterpart along with many other possibilities that ranges from virtual observatory to confidential information transfer in the form of hologram.

## How I built it

Using Unity, Vuforia, Google Vr, Android Sdk and along with the support of google cardboard this app was designed in a way that opens up the world our own world like virtual reality along with augmented reality being turned in to accepted reality. For a while I had to work with the different aspect ratio of the object file respective to the surface and the target photo being properly detected by the camera. The connection and proper conversion of apk files took many trials due to rendering and processing speed of the app. Finally it took me 25 hours to achieve a successful prototype.

## Challenges I ran into

The hardest part was setting up the target image and detecting it. The problem was discovered after 3 hours of brute-forcing. The image used in the creation of target image file has significantly lower resolution than the one printed which led to the recognition of the app become sloppy and sometimes undetectable. Getting the app to work on VR cardboard wasn’t easy. Due to the coagulation of pixels and many other rendering errors and bugs, the output achieved from the simulations(which were perfectly normal) were drastically different form the ones in my android device. The object, most of the time, was hovering half a meter from the point of target image and sometimes was no where to be seen. This was fixed to relocating the image points my changing the co-ordinates.

## Accomplishments that I'm proud of

This is my first hackathon and I can’t believe how hard I have worked for this and finally completed my hack. The first day I was planning on an entirely different hack regarding fingerprint and security but I got in to deep trouble when I realized that hardware collection has fingerprint scanner but not the modules that are absolutely necessary for it to be even used. I had almost given up. I had little, almost none experience in apps development but I started learning the whole concept and idea from scratch slowly. Finally when I realized what I could do with VR hack, I pulled all nighters until I could present something worthy of this hackathon.

## What I learned

This whole week has been very educational. I recently bought macbook and realized how easy it is when I can use terminal in mac, run vuforia exclusively for making augmented reality and on the other hand how I had to pay to use macbook version of adobe 123d catch and windows one was free but most importantly how two finger swipe across the touchpad cost me my whole night of writing a different version of this essay/cover letter. All jokes aside. I learned a lot about app designing, use of different api, learned how to deal with bugs in apps development . Nonetheless I learned about many of the technologies that I never knew existed like galvanic vestibular stimulation.

## What's next for Virtual-Augmented Reality

This projects is merely a prototype compared to the future this idea has. With the upcoming technologies capable of photogrammetry will give rise to higher quality obj file/3D file which if incorporated in augmented reality using camera vr which confuse anyone about what is real or what is fantasy. People can actually see their wildest dreams bring a new era in our society.
## Inspiration

2.2 million deaths were attributed to high blood glucose levels in 2012. However, despite the daunting amount of deaths, glucometer tests are still inconvenient and costly. Test strips cost an astronomical amount and spoil readily in hot and humid weather. These tests will not function in the event of a natural disaster. We decided to change that.

## What it does

Spit into some mud and a user-friendly app will easily analyze the data collected from your saliva sample, quickly telling you if your blood glucose concentration is abnormally high. It's as easy as that.

## How it works

Glucose levels increase in saliva as the glucose concentration in your blood increases. Bacteria eat glucose as food and special bacteria called exoelectrogens (commonly found in mud) can create electricity from it. The more glucose there is in a sample, the higher the voltage. By analyzing the voltage that is produced by the bacteria in the mud, we can determine if your blood glucose levels are dangerously high.

## How we built it

With an Arduino, breadboard, op-amp, and wires, we measured if your glucose levels are either too high (1.0) or healthy (0.0). For software, we used a library called PySerial which can access the Arduino through a serial connection with a laptop. We used pencil lead, a water bottle, and mud to make the glucose sensor.

## Challenges we ran into

We had trouble measuring low voltages using an Arduino. We tried different resistor configurations, attempted to find a voltage sensor, and tried increasing the voltage with an op amp. We resorted to using an op-amp comparator to analyze our voltages.

## Accomplishments that we're proud of

We're proud of blending together all the circuity, software, and biology in this project. Somehow we miraculously made it work.

## What's next for GluClose

The current prototype only determines if your blood glucose level has reached a dangerous level. In the future it'll be able to detect exact glucose concentrations in blood.