Model card

README.md

@@ -0,0 +1,105 @@
+## Emuru Convolutional VAE
+
+This repository hosts the Emuru Convolutional VAE, which is described in our paper. The model features a convolutional Encoder and Decoder with four layers each. The output channels for these layers are 32, 64, 128, and 256, respectively. The Encoder downsamples an input RGB image \(I \in \mathbb{R}^{3 \times W \times H}\) to a latent representation with a single channel and spatial dimensions \(h \times w\) (where \(h = H/8\) and \(w = W/8\)). This design compresses the style information in the image, enabling a lightweight Transformer Decoder to efficiently handle the latent features.
+
+### Training Details
+
+- **Hardware:** NVIDIA RTX 4090  
+- **Iterations:** 60k  
+- **Optimizer:** AdamW with a learning rate of 1e-4  
+- **Loss Components:**  
+  - **MAE Loss (\(\mathcal{L}_{MAE}\))** with weight 1  
+  - **WID Loss (\(\mathcal{L}_{WID}\))** with weight 0.005  
+  - **HTR Loss (\(\mathcal{L}_{HTR}\))** with weight 0.3 (using noisy teacher-forcing with probability 0.3)  
+  - **KL Loss (\(\mathcal{L}_{KL}\))** with weight \(\beta = 1\text{e-6}\)
+
+### Auxiliary Networks
+
+- **Writer Identification:** A ResNet with 6 blocks, trained until achieving 60% accuracy on a synthetic dataset.
+- **Handwritten Text Recognition (HTR):** A Transformer Encoder-Decoder trained until reaching a Character Error Rate (CER) of 0.25 on the synthetic dataset.
+
+### Usage
+
+You can load the pre-trained Emuru VAE using Diffusers’ `AutoencoderKL` interface with a single line of code:
+
+```python
+from diffusers import AutoencoderKL
+model = AutoencoderKL.from_pretrained("vpippi/emuru_vae")
+```
+
+The code snippet below demonstrates how to load an RGB image from disk, encode it into the latent space, decode it back to image space, and save the reconstructed image.
+
+---
+
+### Code Example
+
+```python
+from diffusers import AutoencoderKL
+import torch
+from torchvision.transforms.functional import to_tensor, to_pil_image
+from PIL import Image
+
+# Load the pre-trained Emuru VAE from Hugging Face Hub.
+model = AutoencoderKL.from_pretrained("vpippi/emuru_vae")
+
+# Function to preprocess an RGB image:
+# Loads the image, converts it to RGB, and transforms it to a tensor normalized to [0, 1].
+def preprocess_image(image_path):
+    image = Image.open(image_path).convert("RGB")
+    image_tensor = to_tensor(image).unsqueeze(0)  # Add batch dimension
+    return image_tensor
+
+# Function to postprocess a tensor back to a PIL image for visualization:
+# Clamps the tensor to [0, 1] and converts it to a PIL image.
+def postprocess_tensor(tensor):
+    tensor = torch.clamp(tensor, 0, 1).squeeze(0)  # Remove batch dimension
+    return to_pil_image(tensor)
+
+# Example: Encode and decode an image.
+# Replace with your image path.
+image_path = "/path/to/image"
+input_image = preprocess_image(image_path)
+
+# Encode the image to the latent space.
+# The encode() method returns an object with a 'latent_dist' attribute.
+# We sample from this distribution to obtain the latent representation.
+with torch.no_grad():
+    latent_dist = model.encode(input_image).latent_dist
+    latents = latent_dist.sample()
+
+# Decode the latent representation back to image space.
+with torch.no_grad():
+    reconstructed = model.decode(latents).sample
+
+# Load the original image for comparison.
+original_image = Image.open(image_path).convert("RGB")
+# Convert the reconstructed tensor back to a PIL image.
+reconstructed_image = postprocess_tensor(reconstructed)
+
+# Save the reconstructed image.
+reconstructed_image.save("reconstructed_image.png")
+```
+
+---
+
+### Testing with Sample Images
+
+If you wish to test the model on sample images hosted on the Hugging Face Hub, you can:
+- **Include sample images in your repository:** Place images in a folder (e.g., `samples/`) and reference them in your code.
+- **Use the `huggingface_hub` API:** Download images programmatically using the `hf_hub_download` function.
+
+For example, to download a sample image from your repository:
+
+```python
+from huggingface_hub import hf_hub_download
+from PIL import Image
+
+# Replace 'vpippi/emuru_vae' and 'samples/lam_sample.jpg' with your details.
+image_path = hf_hub_download(repo_id="vpippi/emuru_vae", filename="samples/lam_sample.jpg")
+sample_image = Image.open(image_path).convert("RGB")
+sample_image.show()
+```
+
+This approach allows you to easily test and demonstrate the capabilities of the Emuru VAE using images hosted on the Hugging Face Hub.
+
+Feel free to modify the preprocessing steps to suit your needs. Enjoy experimenting with the Emuru VAE!