Creating your custom Ghibli Text-to-Image model

TL;DR

While Ghibli-style AI-generated images have become a global trend, we're taking a unique approach by creating something more culturally meaningful. We've developed a specialized Text-to-Image model fine-tuned from the Flux.1-dev diffusion model using LoRA, trained on a carefully curated dataset that celebrates Moroccan heritage.

Simply input a prompt and watch as Moroccan cultural elements blend seamlessly with Ghibli's iconic aesthetic, all powered by state-of-the-art AI technology.

Introduction

Studio Ghibli's distinctive art style has captured imaginations worldwide, inspiring creators to reimagine diverse scenes through its enchanting lens. From bustling cityscapes to mystical realms, the Ghibli aesthetic transforms ordinary scenes into magical vignettes filled with warmth and wonder.

However, we identified two significant challenges in the current landscape:

• Accessibility Issues: Popular tools like ChatGPT face limitations with free-tier usage caps and server availability, making consistent access challenging.

• Cultural Authenticity: Existing models often misrepresent Moroccan cultural elements. Whether it's incorrectly depicted mosque architecture or misplaced traditional attire, these tools frequently fall short in authentically capturing Morocco's rich cultural heritage.

To address these challenges, we've developed our own specialized model that combines Ghibli's artistic style with an authentic understanding of Moroccan culture. Best of all, we're making both the code and weights open-source for the community. Try It Out! Experience the model yourself at our Hugging Face Space: https://huggingface.co/spaces/atlasia/flux_moroccan_ghibli_style

Our Approach

Dataset Creation

We carefully crafted 11 prompts, each designed to capture distinct elements of Moroccan culture. Our selection criteria focused on:

Authentic cultural representation (e.g., zelij craftsmen in traditional souks) Concise yet descriptive prompting Flexibility for artistic interpretation within the Ghibli aesthetic

The dataset is available at atlasia/Ghibli-style-morocco-dataset and includes diverse scenes such as:

• Traditional riad courtyards
• Medina street scenes
• Zellige craftsmen in souks
• Hassan II Mosque
• Traditional family scenes
• Cultural ceremonies

Our unique approach involved:

Curating culturally authentic Moroccan-themed prompts Utilizing ChatGPT to reimagine these scenes in Ghibli's distinctive style

Training Methodology

We leveraged ai-toolkit for our training pipeline, chosen for its user-friendly interface and robust capabilities.

Key training parameters:

• Base Model: FLUX.1-dev (Black Forest Labs)
• Approach: LoRA-based fine-tuning
• Dataset: 11 carefully curated images with corresponding captions
• Dropout Rate: 5% for variation
• Image Resolutions: Multi-scale (512×512, 768×768, 1024×1024)
• Training Steps: 4000
• Batch Size: 1
• Optimizer: AdamW (learning rate: 4e-4)
• Noise Scheduler: flowmatch
• Architecture: UNet training with frozen text encoder

The model is available at atlasia/moroccan-ghibli-flux-lora with multiple checkpoints.

Validation Process

We conducted validation every 250 steps using fixed prompts to track progress:

Initial State (Step 0): The model produced realistic but stylistically inconsistent images, lacking Ghibli's characteristic dreamlike quality.

Early Progress (Step 250): Rapid improvement in style adaptation, showing promising results.

Later Stages: Model began showing signs of overfitting in later steps.

Using the Model

To use our model, you'll need to include the trigger word "Moroccan Ghibli studio style" in your prompts. Here's a quick example:

import torch
from diffusers import FluxPipeline
from huggingface_hub import hf_hub_download

# Load base model
pipe = FluxPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev", 
    torch_dtype=torch.bfloat16
)
# Load LoRA weights
lora_path = hf_hub_download(
    repo_id="atlasia/moroccan-ghibli-flux-lora",
    filename="moroccan_ghibli_flux_lora_000001250.safetensors",
    revision="step_1250"
)
pipe.load_lora_weights(lora_path)

# Generate image
prompt = "Moroccan Ghibli studio style traditional riad courtyard"
image = pipe(
    prompt,
    height=1024,
    width=1024,
    guidance_scale=3.5,
    num_inference_steps=50
).images[0]

Example prompts that work well:

• "Moroccan Ghibli studio style of a traditional riad courtyard with a fountain": "Moroccan Ghibli studio style of a medina street scene at sunset":
• "Moroccan Ghibli studio style of a traditional tea ceremony":
• "Moroccan Ghibli studio style of an ancient palace interior"

Conclusion

Our findings demonstrate that even with minimal fine-tuning, models can produce culturally rich and stylistically coherent images. This suggests that carefully curated, small-scale datasets can yield significant results, opening doors for more targeted and efficient model training.

Ethical Considerations

We acknowledge that all artistic credit belongs to Hayao Miyazaki and Studio Ghibli. Our project is purely research-driven, aiming to study the intersection of culture, style, and machine learning. We've open-sourced our model and weights to support the research community while firmly opposing any unauthorized commercial use of artists' styles.

Future Directions

Our roadmap includes:

• Expanding our dataset with diverse Moroccan scenes
• Exploring additional animation styles while respecting artistic integrity
• Implementing style strength controls
• Creating comprehensive fine-tuning tutorials

Acknowledgments

We extend our gratitude to all project collaborators: Anas Amchaar, Nouamane Tazi, Abdellah Oumida, Mohammed Sbaihi, Abdeljalil Elmajjodi, Zaid Chiech, Ibtissam Achaoui, Oumayma Essarhi, Abdelaziz Guelfane.

A special thanks to Hugging Face for sponsoring our community.

Join Our Community

• Website: https://www.atlasia.ma/
• Hugging Face: https://huggingface.co/atlasia

Citation

@article{atlasia2025Ghibli_Model_v1,
  title={Creating your custom Ghibli Text-to-Image model},
  author={Amchaar Anas, Oumida Abdellah, Abdeljalil Elmajjodi},
  year={2025},
  url={https://huggingface.co/spaces/atlasia/flux_moroccan_ghibli_style}
  organization={AtlasIA}
}