README.md

metadata

datasets:
  - Eunju2834/img_captioning_oilcanvas_style
  - Eunju2834/oil_impressionism_style
pipeline_tag: text-to-image
base_model: runwayml/stable-diffusion-v1-5
tags:
  - LoRA
  - Diffusion
  - OilCanvas
  - stable-diffusion
  - text-to-image
license: creativeml-openrail-m

🎨 LoRA text2image fine-tuning - Oil Canvas Style

📄 Model Description

This model is a fine-tuned version of the Stable Diffusion v1.5 model using Low-Rank Adaptation (LoRA) techniques to generate images in an oil canvas painting style, specifically focusing on the Impressionism genre. The model is designed to produce vibrant, brushstroke-rich images with a joyful and community-focused theme.

🔍 Model Details

• Base Model: Stable Diffusion v1.5 (runwayml/stable-diffusion-v1-5)
• Fine-Tuning Method: LoRA (Low-Rank Adaptation)
• Training Data: Custom oil canvas style dataset collected from Kaggle, focused on Impressionism artworks (Eunju2834/img_captioning_oilcanvas_style,Eunju2834/oil_impressionism_style)
• Captioning Method: BLIP-2 model used to generate image captions for the dataset
• Training Configuration:
  • Epochs: 20
  • Batch Size: 1
  • Learning Rate: 1e-4
  • Scheduler: Cosine
  • Seed: 2024

🚀 Usage

from diffusers import StableDiffusionPipeline
import torch

model_path = 'Eunju2834/LoRA_oilcanvas_style'
pipe = StableDiffusionPipeline.from_pretrained('runwayml/stable-diffusion-v1-5', torch_dtype=torch.float16, use_auth_token=True)
pipe.unet.load_attn_procs(model_path)
pipe.to('cuda')

prompt = '''(Oil Painting: 1.1), (Impressionism: 1.2), (oil painting with brush strokes: 1.2),
Park stroll, joyful atmosphere, laughter-filled time, playful dogs, vibrant park scene,
cheerful interactions, happy pet owners, heartwarming moments, vibrant community vibes'''

neg_prompt = '''FastNegativeV2, (bad-artist:1.0), (worst quality, low quality:1.4),
(watermark), error, missing fingers, extra digit, cropped, normal quality, blurry'''

image = pipe(prompt, negative_prompt=neg_prompt, num_inference_steps=30, guidance_scale=7.5).images[0]
image.save('oil_impressionism_park_stroll.png')

🖼️ Example Results

The model generates images like the ones below, showcasing an oil painting style with vibrant colors and Impressionist influences.

⚠️ Limitations and Biases

• The model is optimized for oil canvas style images and may not generalize well to other artistic styles.
• Potential biases may exist due to the specific nature of the training dataset (e.g., Impressionism artworks).

📜 License

CreativeML Open RAIL-M