Emuru

Emuru is a conditional generative model that integrates a T5-based decoder with a Variational Autoencoder (VAE) for image generation conditioned on text and style images. It allows users to combine textual prompts (e.g., style text, generation text) and style images to create new, synthesized images.

Model description

• Architecture:
  Emuru uses a T5ForConditionalGeneration as its text decoder and an AutoencoderKL as the VAE backbone. The T5 model encodes textual prompts and partially decoded latent representations, then predicts the next latent tokens. The VAE is used both to encode the initial style image and decode the predicted latent tokens back into an image.

• Inputs:

  1. Style Image: A reference image, which Emuru encodes to capture its “style” or other visual characteristics.
  2. Style Text: Text describing the style or context.
  3. Generation Text: Text describing the content or object to generate.

• Outputs:

  1. A synthesized image that reflects the fused style and text descriptions.

• Tokenization:
  Emuru uses AutoTokenizer to handle the text prompts, which adjusts the T5’s vocabulary and token embeddings accordingly.

• Usage scenarios:

  • Stylized text-to-image generation
  • Image manipulation or design prototyping based on textual descriptions
  • Research or educational demonstrations of T5-based generative modeling

How to use

Below is a minimal usage example in Python. You can load the model with AutoModel.from_pretrained(...) and simply call .generate(...) or .generate_batch(...) to create images.

import torch
from PIL import Image
from transformers import AutoModel
from huggingface_hub import hf_hub_download
from torchvision.transforms import functional as F

def load_image(img_path):
    img = Image.open(img_path).convert("RGB")
    # Resize the image to have a fixed height of 64 pixels
    img = img.resize((img.width * 64 // img.height, 64))
    img = F.to_tensor(img)
    img = F.normalize(img, [0.5], [0.5])
    return img

# 1. Load the model
model = AutoModel.from_pretrained("blowing-up-groundhogs/emuru", trust_remote_code=True)
model.cuda()  # Move to GPU if available

# 2. Prepare your inputs
style_text = 'THE JOLLY IS "U"'
gen_text = 'EMURU'
img_path = hf_hub_download(repo_id="blowing-up-groundhogs/emuru", filename="sample.png")
style_img = load_image(img_path)
style_img = style_img.cuda()

# 3. Generate an image
generated_pil_image = model.generate(
    style_text=style_text,
    gen_text=gen_text,
    style_img=style_img,
    max_new_tokens=64
)

# 4. Save the result
generated_pil_image.save("generated_image.png")

Batch Generation

You can also generate a batch of images if you have multiple style texts, generation texts, and style images:

style_texts = ['THE JOLLY IS "U"', 'THE JOLLY IS "M"', 'THE JOLLY IS "R"']
gen_texts   = ['EMURU', 'EMURU', 'EMURU']
style_imgs  = torch.stack([style_img, style_img, style_img], dim=0)  # shape: (batch_size, C, H, W)
lengths     = [style_img.size(-1), style_img.size(-1), style_img.size(-1)]

output_images = model.generate_batch(
    style_texts=style_texts,
    gen_texts=gen_texts,
    style_imgs=style_imgs,
    lengths=lengths,
    max_new_tokens=64
)

# `output_images` is a list of PIL images
for idx, pil_img in enumerate(output_images):
    pil_img.save(f"batch_generated_image_{idx}.png")

Citation

If you use Emuru in your research or wish to refer to it, please cite:

Wait for it...