metadata

language:
  - en
tags:
  - image-generation
  - text-to-image
  - vae
  - t5
  - conditional-generation
  - generative-modeling
  - image-synthesis
  - image-manipulation
  - design-prototyping
  - research
  - educational
license: mit
datasets:
  - blowing-up-groundhogs/font-square-v2
metrics:
  - FID
  - KID
  - HWD
  - CER
library_name: t5

Emuru

Emuru is a conditional generative model that integrates a T5-based decoder with a Variational Autoencoder (VAE) for image generation conditioned on text and style images. It allows users to combine textual prompts (e.g., style text, generation text) and style images to create new, synthesized images.

Model description

Architecture:
Emuru uses a T5ForConditionalGeneration as its text decoder and an AutoencoderKL as the VAE backbone. The T5 model encodes textual prompts and partially decoded latent representations, then predicts the next latent tokens. The VAE is used both to encode the initial style image and decode the predicted latent tokens back into an image.
Inputs:
1. Style Image: A reference image, which Emuru encodes to capture its “style” or other visual characteristics.
2. Style Text: Text describing the style or context.
3. Generation Text: Text describing the content or object to generate.
Outputs:
1. A synthesized image that reflects the fused style and text descriptions.
Tokenization:
Emuru uses AutoTokenizer to handle the text prompts, which adjusts the T5’s vocabulary and token embeddings accordingly.
Usage scenarios:
- Stylized text-to-image generation
- Image manipulation or design prototyping based on textual descriptions
- Research or educational demonstrations of T5-based generative modeling

How to use

Below is a minimal usage example in Python. You can load the model with AutoModel.from_pretrained(...) and simply call .generate(...) or .generate_batch(...) to create images.

import torch
from PIL import Image
from transformers import AutoModel
from torchvision.transforms import functional as F

# 1. Load the model
model = AutoModel.from_pretrained("blowing-up-groundhogs/emuru")
model.cuda()  # Move to GPU if available

# 2. Prepare your inputs
style_text = "A beautiful watercolor style"
gen_text = "A majestic mountain with a rainbow"
style_img = Image.open("my_style_image.png").convert("RGB")

# Convert the style image to a suitable tensor
style_img = F.to_tensor(style_img)
style_img = F.resize((style_img.width * 64 // style_img.height, 64))    # Example resize
style_img = F.normalize(style_img, [0.5], [0.5])  # Normalize to [-1, 1]
style_img = style_img.unsqueeze(0).cuda()

# 3. Generate an image
generated_pil_image = model.generate(
    style_text=style_text,
    gen_text=gen_text,
    style_img=style_img,
    max_new_tokens=64
)

# 4. Save or display the result
generated_pil_image.save("generated_image.png")

Batch Generation

You can also generate a batch of images if you have multiple style texts, generation texts, and style images:

style_texts = ["Style text 1", "Style text 2"]
gen_texts   = ["Gen text 1", "Gen text 2"]
style_imgs  = torch.stack([img1, img2], dim=0)  # shape: (batch_size, C, H, W)
lengths     = [img1.size(-1), img2.size(-1)]

output_images = model.generate_batch(
    style_texts=style_texts,
    gen_texts=gen_texts,
    style_imgs=style_imgs,
    lengths=lengths,
    max_new_tokens=64
)

# `output_images` is a list of PIL images
for idx, pil_img in enumerate(output_images):
    pil_img.save(f"batch_generated_image_{idx}.png")

Citation

If you use Emuru in your research or wish to refer to it, please cite:

...