|
--- |
|
datasets: |
|
- Eunju2834/img_captioning_oilcanvas_style |
|
- Eunju2834/oil_impressionism_style |
|
pipeline_tag: text-to-image |
|
base_model: runwayml/stable-diffusion-v1-5 |
|
tags: |
|
- LoRA |
|
- Diffusion |
|
- OilCanvas |
|
- stable-diffusion |
|
- text-to-image |
|
license: creativeml-openrail-m |
|
--- |
|
|
|
# 🎨 LoRA text2image fine-tuning - Oil Canvas Style |
|
|
|
## 📄 Model Description |
|
This model is a fine-tuned version of the Stable Diffusion v1.5 model using Low-Rank Adaptation (LoRA) techniques to generate images in an oil canvas painting style, specifically focusing on the Impressionism genre. The model is designed to produce vibrant, brushstroke-rich images with a joyful and community-focused theme. |
|
|
|
## 🔍 Model Details |
|
- **Base Model:** Stable Diffusion v1.5 (`runwayml/stable-diffusion-v1-5`) |
|
- **Fine-Tuning Method:** LoRA (Low-Rank Adaptation) |
|
- **Training Data:** Custom oil canvas style dataset collected from Kaggle, focused on Impressionism artworks (`Eunju2834/img_captioning_oilcanvas_style`,`Eunju2834/oil_impressionism_style`) |
|
- **Captioning Method:** BLIP-2 model used to generate image captions for the dataset |
|
- **Training Configuration:** |
|
- Epochs: 20 |
|
- Batch Size: 1 |
|
- Learning Rate: 1e-4 |
|
- Scheduler: Cosine |
|
- Seed: 2024 |
|
|
|
## 🚀 Usage |
|
```python |
|
from diffusers import StableDiffusionPipeline |
|
import torch |
|
|
|
model_path = 'Eunju2834/LoRA_oilcanvas_style' |
|
pipe = StableDiffusionPipeline.from_pretrained('runwayml/stable-diffusion-v1-5', torch_dtype=torch.float16, use_auth_token=True) |
|
pipe.unet.load_attn_procs(model_path) |
|
pipe.to('cuda') |
|
|
|
prompt = '''(Oil Painting: 1.1), (Impressionism: 1.2), (oil painting with brush strokes: 1.2), |
|
Park stroll, joyful atmosphere, laughter-filled time, playful dogs, vibrant park scene, |
|
cheerful interactions, happy pet owners, heartwarming moments, vibrant community vibes''' |
|
|
|
neg_prompt = '''FastNegativeV2, (bad-artist:1.0), (worst quality, low quality:1.4), |
|
(watermark), error, missing fingers, extra digit, cropped, normal quality, blurry''' |
|
|
|
image = pipe(prompt, negative_prompt=neg_prompt, num_inference_steps=30, guidance_scale=7.5).images[0] |
|
image.save('oil_impressionism_park_stroll.png') |
|
``` |
|
|
|
## 🖼️ Example Results |
|
The model generates images like the ones below, showcasing an oil painting style with vibrant colors and Impressionist influences. |
|
<p style="display: flex; justify-content: center;"> |
|
<img src="https://cdn-uploads.huggingface.co/production/uploads/64bad345f671da974eeb1ba3/SfjQLeVKie31DZSB23jrq.png" width="250px" /> |
|
<img src="https://cdn-uploads.huggingface.co/production/uploads/64bad345f671da974eeb1ba3/sUTDFT20kGhrRUilSgVOe.png" width="250px" /> |
|
<img src="https://cdn-uploads.huggingface.co/production/uploads/64bad345f671da974eeb1ba3/MhGz7DPm_Eb_NWoIxQx28.png" width="250px" /> |
|
</p> |
|
|
|
|
|
|
|
## ⚠️ Limitations and Biases |
|
- The model is optimized for oil canvas style images and may not generalize well to other artistic styles. |
|
- Potential biases may exist due to the specific nature of the training dataset (e.g., Impressionism artworks). |
|
|
|
## 📜 License |
|
CreativeML Open RAIL-M |
|
|
|
|
|
|
|
|
