File size: 10,352 Bytes
20bf0a0
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
<!--Copyright 2024 The HuggingFace Team. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.
-->


# Text-to-image

<Tip warning={true}>

text-to-image νŒŒμΈνŠœλ‹ μŠ€ν¬λ¦½νŠΈλŠ” experimental μƒνƒœμž…λ‹ˆλ‹€. κ³Όμ ν•©ν•˜κΈ° 쉽고 치λͺ…적인 망각과 같은 λ¬Έμ œμ— λΆ€λ”ͺ히기 μ‰½μŠ΅λ‹ˆλ‹€. 자체 λ°μ΄ν„°μ…‹μ—μ„œ μ΅œμƒμ˜ κ²°κ³Όλ₯Ό μ–»μœΌλ €λ©΄ λ‹€μ–‘ν•œ ν•˜μ΄νΌνŒŒλΌλ―Έν„°λ₯Ό νƒμƒ‰ν•˜λŠ” 것이 μ’‹μŠ΅λ‹ˆλ‹€.

</Tip>

Stable Diffusionκ³Ό 같은 text-to-image λͺ¨λΈμ€ ν…μŠ€νŠΈ ν”„λ‘¬ν”„νŠΈμ—μ„œ 이미지λ₯Ό μƒμ„±ν•©λ‹ˆλ‹€. 이 κ°€μ΄λ“œλŠ” PyTorch 및 Flaxλ₯Ό μ‚¬μš©ν•˜μ—¬ 자체 λ°μ΄ν„°μ…‹μ—μ„œ [`CompVis/stable-diffusion-v1-4`](https://huggingface.co/CompVis/stable-diffusion-v1-4) λͺ¨λΈλ‘œ νŒŒμΈνŠœλ‹ν•˜λŠ” 방법을 λ³΄μ—¬μ€λ‹ˆλ‹€. 이 κ°€μ΄λ“œμ— μ‚¬μš©λœ text-to-image νŒŒμΈνŠœλ‹μ„ μœ„ν•œ λͺ¨λ“  ν•™μŠ΅ μŠ€ν¬λ¦½νŠΈμ— 관심이 μžˆλŠ” 경우 이 [리포지토리](https://github.com/huggingface/diffusers/tree/main/examples/text_to_image)μ—μ„œ μžμ„Ένžˆ 찾을 수 μžˆμŠ΅λ‹ˆλ‹€.

슀크립트λ₯Ό μ‹€ν–‰ν•˜κΈ° 전에, 라이브러리의 ν•™μŠ΅ dependency듀을 μ„€μΉ˜ν•΄μ•Ό ν•©λ‹ˆλ‹€:

```bash
pip install git+https://github.com/huggingface/diffusers.git
pip install -U -r requirements.txt
```

그리고 [πŸ€—Accelerate](https://github.com/huggingface/accelerate/) ν™˜κ²½μ„ μ΄ˆκΈ°ν™”ν•©λ‹ˆλ‹€:

```bash
accelerate config
```

리포지토리λ₯Ό 이미 λ³΅μ œν•œ 경우, 이 단계λ₯Ό μˆ˜ν–‰ν•  ν•„μš”κ°€ μ—†μŠ΅λ‹ˆλ‹€. λŒ€μ‹ , 둜컬 체크아웃 경둜λ₯Ό ν•™μŠ΅ μŠ€ν¬λ¦½νŠΈμ— λͺ…μ‹œν•  수 있으며 κ±°κΈ°μ—μ„œ λ‘œλ“œλ©λ‹ˆλ‹€.

### ν•˜λ“œμ›¨μ–΄ μš”κ΅¬ 사항

`gradient_checkpointing` 및 `mixed_precision`을 μ‚¬μš©ν•˜λ©΄ 단일 24GB GPUμ—μ„œ λͺ¨λΈμ„ νŒŒμΈνŠœλ‹ν•  수 μžˆμŠ΅λ‹ˆλ‹€. 더 높은 `batch_size`와 더 λΉ λ₯Έ ν›ˆλ ¨μ„ μœ„ν•΄μ„œλŠ” GPU λ©”λͺ¨λ¦¬κ°€ 30GB 이상인 GPUλ₯Ό μ‚¬μš©ν•˜λŠ” 것이 μ’‹μŠ΅λ‹ˆλ‹€. TPU λ˜λŠ” GPUμ—μ„œ νŒŒμΈνŠœλ‹μ„ μœ„ν•΄ JAXλ‚˜ Flaxλ₯Ό μ‚¬μš©ν•  μˆ˜λ„ μžˆμŠ΅λ‹ˆλ‹€. μžμ„Έν•œ λ‚΄μš©μ€ [μ•„λž˜](#flax-jax-finetuning)λ₯Ό μ°Έμ‘°ν•˜μ„Έμš”.

xFormers둜 memory efficient attention을 ν™œμ„±ν™”ν•˜μ—¬ λ©”λͺ¨λ¦¬ μ‚¬μš©λŸ‰ 훨씬 더 쀄일 수 μžˆμŠ΅λ‹ˆλ‹€. [xFormersκ°€ μ„€μΉ˜](./optimization/xformers)λ˜μ–΄ μžˆλŠ”μ§€ ν™•μΈν•˜κ³  `--enable_xformers_memory_efficient_attention`λ₯Ό ν•™μŠ΅ μŠ€ν¬λ¦½νŠΈμ— λͺ…μ‹œν•©λ‹ˆλ‹€.

xFormersλŠ” Flax에 μ‚¬μš©ν•  수 μ—†μŠ΅λ‹ˆλ‹€.

## Hub에 λͺ¨λΈ μ—…λ‘œλ“œν•˜κΈ°

ν•™μŠ΅ μŠ€ν¬λ¦½νŠΈμ— λ‹€μŒ 인수λ₯Ό μΆ”κ°€ν•˜μ—¬ λͺ¨λΈμ„ ν—ˆλΈŒμ— μ €μž₯ν•©λ‹ˆλ‹€:

```bash
  --push_to_hub
```


## 체크포인트 μ €μž₯ 및 뢈러였기

ν•™μŠ΅ 쀑 λ°œμƒν•  수 μžˆλŠ” 일에 λŒ€λΉ„ν•˜μ—¬ μ •κΈ°μ μœΌλ‘œ 체크포인트λ₯Ό μ €μž₯ν•΄ λ‘λŠ” 것이 μ’‹μŠ΅λ‹ˆλ‹€. 체크포인트λ₯Ό μ €μž₯ν•˜λ €λ©΄ ν•™μŠ΅ μŠ€ν¬λ¦½νŠΈμ— λ‹€μŒ 인수λ₯Ό λͺ…μ‹œν•©λ‹ˆλ‹€.

```bash
  --checkpointing_steps=500
```

500μŠ€ν…λ§ˆλ‹€ 전체 ν•™μŠ΅ stateκ°€ 'output_dir'의 ν•˜μœ„ 폴더에 μ €μž₯λ©λ‹ˆλ‹€. μ²΄ν¬ν¬μΈνŠΈλŠ” 'checkpoint-'에 μ§€κΈˆκΉŒμ§€ ν•™μŠ΅λœ step μˆ˜μž…λ‹ˆλ‹€. 예λ₯Ό λ“€μ–΄ 'checkpoint-1500'은 1500 ν•™μŠ΅ step 후에 μ €μž₯된 μ²΄ν¬ν¬μΈνŠΈμž…λ‹ˆλ‹€.

ν•™μŠ΅μ„ μž¬κ°œν•˜κΈ° μœ„ν•΄ 체크포인트λ₯Ό 뢈러였렀면 '--resume_from_checkpoint' 인수λ₯Ό ν•™μŠ΅ μŠ€ν¬λ¦½νŠΈμ— λͺ…μ‹œν•˜κ³  μž¬κ°œν•  체크포인트λ₯Ό μ§€μ •ν•˜μ‹­μ‹œμ˜€. 예λ₯Ό λ“€μ–΄ λ‹€μŒ μΈμˆ˜λŠ” 1500개의 ν•™μŠ΅ step 후에 μ €μž₯된 μ²΄ν¬ν¬μΈνŠΈμ—μ„œλΆ€ν„° ν›ˆλ ¨μ„ μž¬κ°œν•©λ‹ˆλ‹€.

```bash
  --resume_from_checkpoint="checkpoint-1500"
```

## νŒŒμΈνŠœλ‹

<frameworkcontent>
<pt>
λ‹€μŒκ³Ό 같이 [Naruto BLIP μΊ‘μ…˜](https://huggingface.co/datasets/lambdalabs/naruto-blip-captions) λ°μ΄ν„°μ…‹μ—μ„œ νŒŒμΈνŠœλ‹ 싀행을 μœ„ν•΄ [PyTorch ν•™μŠ΅ 슀크립트](https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image.py)λ₯Ό μ‹€ν–‰ν•©λ‹ˆλ‹€:


```bash
export MODEL_NAME="CompVis/stable-diffusion-v1-4"
export dataset_name="lambdalabs/naruto-blip-captions"

accelerate launch train_text_to_image.py \
  --pretrained_model_name_or_path=$MODEL_NAME \
  --dataset_name=$dataset_name \
  --use_ema \
  --resolution=512 --center_crop --random_flip \
  --train_batch_size=1 \
  --gradient_accumulation_steps=4 \
  --gradient_checkpointing \
  --mixed_precision="fp16" \
  --max_train_steps=15000 \
  --learning_rate=1e-05 \
  --max_grad_norm=1 \
  --lr_scheduler="constant" --lr_warmup_steps=0 \
  --output_dir="sd-naruto-model"
```

자체 λ°μ΄ν„°μ…‹μœΌλ‘œ νŒŒμΈνŠœλ‹ν•˜λ €λ©΄ πŸ€— [Datasets](https://huggingface.co/docs/datasets/index)μ—μ„œ μš”κ΅¬ν•˜λŠ” ν˜•μ‹μ— 따라 데이터셋을 μ€€λΉ„ν•˜μ„Έμš”. [데이터셋을 ν—ˆλΈŒμ— μ—…λ‘œλ“œ](https://huggingface.co/docs/datasets/image_dataset#upload-dataset-to-the-hub)ν•˜κ±°λ‚˜ [νŒŒμΌλ“€μ΄ μžˆλŠ” 둜컬 폴더λ₯Ό μ€€λΉ„](https ://huggingface.co/docs/datasets/image_dataset#imagefolder)ν•  수 μžˆμŠ΅λ‹ˆλ‹€.

μ‚¬μš©μž μ»€μŠ€ν…€ loading logic을 μ‚¬μš©ν•˜λ €λ©΄ 슀크립트λ₯Ό μˆ˜μ •ν•˜μ‹­μ‹œμ˜€. 도움이 λ˜λ„λ‘ μ½”λ“œμ˜ μ μ ˆν•œ μœ„μΉ˜μ— 포인터λ₯Ό λ‚¨κ²ΌμŠ΅λ‹ˆλ‹€. πŸ€— μ•„λž˜ 예제 μŠ€ν¬λ¦½νŠΈλŠ” `TRAIN_DIR`의 둜컬 λ°μ΄ν„°μ…‹μœΌλ‘œλ₯Ό νŒŒμΈνŠœλ‹ν•˜λŠ” 방법과 `OUTPUT_DIR`μ—μ„œ λͺ¨λΈμ„ μ €μž₯ν•  μœ„μΉ˜λ₯Ό λ³΄μ—¬μ€λ‹ˆλ‹€:


```bash
export MODEL_NAME="CompVis/stable-diffusion-v1-4"
export TRAIN_DIR="path_to_your_dataset"
export OUTPUT_DIR="path_to_save_model"

accelerate launch train_text_to_image.py \
  --pretrained_model_name_or_path=$MODEL_NAME \
  --train_data_dir=$TRAIN_DIR \
  --use_ema \
  --resolution=512 --center_crop --random_flip \
  --train_batch_size=1 \
  --gradient_accumulation_steps=4 \
  --gradient_checkpointing \
  --mixed_precision="fp16" \
  --max_train_steps=15000 \
  --learning_rate=1e-05 \
  --max_grad_norm=1 \
  --lr_scheduler="constant" --lr_warmup_steps=0 \
  --output_dir=${OUTPUT_DIR}
```

</pt>
<jax>
[@duongna211](https://github.com/duongna21)의 κΈ°μ—¬λ‘œ, Flaxλ₯Ό μ‚¬μš©ν•΄ TPU 및 GPUμ—μ„œ Stable Diffusion λͺ¨λΈμ„ 더 λΉ λ₯΄κ²Œ ν•™μŠ΅ν•  수 μžˆμŠ΅λ‹ˆλ‹€. μ΄λŠ” TPU ν•˜λ“œμ›¨μ–΄μ—μ„œ 맀우 νš¨μœ¨μ μ΄μ§€λ§Œ GPUμ—μ„œλ„ ν›Œλ₯­ν•˜κ²Œ μž‘λ™ν•©λ‹ˆλ‹€. Flax ν•™μŠ΅ μŠ€ν¬λ¦½νŠΈλŠ” gradient checkpointingλ‚˜ gradient accumulationκ³Ό 같은 κΈ°λŠ₯을 아직 μ§€μ›ν•˜μ§€ μ•ŠμœΌλ―€λ‘œ λ©”λͺ¨λ¦¬κ°€ 30GB 이상인 GPU λ˜λŠ” TPU v3κ°€ ν•„μš”ν•©λ‹ˆλ‹€.

슀크립트λ₯Ό μ‹€ν–‰ν•˜κΈ° 전에 μš”κ΅¬ 사항이 μ„€μΉ˜λ˜μ–΄ μžˆλŠ”μ§€ ν™•μΈν•˜μ‹­μ‹œμ˜€:

```bash
pip install -U -r requirements_flax.txt
```

그러면 λ‹€μŒκ³Ό 같이 [Flax ν•™μŠ΅ 슀크립트](https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image_flax.py)λ₯Ό μ‹€ν–‰ν•  수 μžˆμŠ΅λ‹ˆλ‹€.

```bash
export MODEL_NAME="stable-diffusion-v1-5/stable-diffusion-v1-5"
export dataset_name="lambdalabs/naruto-blip-captions"

python train_text_to_image_flax.py \
  --pretrained_model_name_or_path=$MODEL_NAME \
  --dataset_name=$dataset_name \
  --resolution=512 --center_crop --random_flip \
  --train_batch_size=1 \
  --max_train_steps=15000 \
  --learning_rate=1e-05 \
  --max_grad_norm=1 \
  --output_dir="sd-naruto-model"
```

자체 λ°μ΄ν„°μ…‹μœΌλ‘œ νŒŒμΈνŠœλ‹ν•˜λ €λ©΄ πŸ€— [Datasets](https://huggingface.co/docs/datasets/index)μ—μ„œ μš”κ΅¬ν•˜λŠ” ν˜•μ‹μ— 따라 데이터셋을 μ€€λΉ„ν•˜μ„Έμš”. [데이터셋을 ν—ˆλΈŒμ— μ—…λ‘œλ“œ](https://huggingface.co/docs/datasets/image_dataset#upload-dataset-to-the-hub)ν•˜κ±°λ‚˜ [νŒŒμΌλ“€μ΄ μžˆλŠ” 둜컬 폴더λ₯Ό μ€€λΉ„](https ://huggingface.co/docs/datasets/image_dataset#imagefolder)ν•  수 μžˆμŠ΅λ‹ˆλ‹€.

μ‚¬μš©μž μ»€μŠ€ν…€ loading logic을 μ‚¬μš©ν•˜λ €λ©΄ 슀크립트λ₯Ό μˆ˜μ •ν•˜μ‹­μ‹œμ˜€. 도움이 λ˜λ„λ‘ μ½”λ“œμ˜ μ μ ˆν•œ μœ„μΉ˜μ— 포인터λ₯Ό λ‚¨κ²ΌμŠ΅λ‹ˆλ‹€. πŸ€— μ•„λž˜ 예제 μŠ€ν¬λ¦½νŠΈλŠ” `TRAIN_DIR`의 둜컬 λ°μ΄ν„°μ…‹μœΌλ‘œλ₯Ό νŒŒμΈνŠœλ‹ν•˜λŠ” 방법을 λ³΄μ—¬μ€λ‹ˆλ‹€:

```bash
export MODEL_NAME="duongna/stable-diffusion-v1-4-flax"
export TRAIN_DIR="path_to_your_dataset"

python train_text_to_image_flax.py \
  --pretrained_model_name_or_path=$MODEL_NAME \
  --train_data_dir=$TRAIN_DIR \
  --resolution=512 --center_crop --random_flip \
  --train_batch_size=1 \
  --mixed_precision="fp16" \
  --max_train_steps=15000 \
  --learning_rate=1e-05 \
  --max_grad_norm=1 \
  --output_dir="sd-naruto-model"
```
</jax>
</frameworkcontent>

## LoRA

Text-to-image λͺ¨λΈ νŒŒμΈνŠœλ‹μ„ μœ„ν•΄, λŒ€κ·œλͺ¨ λͺ¨λΈ ν•™μŠ΅μ„ κ°€μ†ν™”ν•˜κΈ° μœ„ν•œ νŒŒμΈνŠœλ‹ 기술인 LoRA(Low-Rank Adaptation of Large Language Models)λ₯Ό μ‚¬μš©ν•  수 μžˆμŠ΅λ‹ˆλ‹€. μžμ„Έν•œ λ‚΄μš©μ€ [LoRA ν•™μŠ΅](lora#text-to-image) κ°€μ΄λ“œλ₯Ό μ°Έμ‘°ν•˜μ„Έμš”.

## μΆ”λ‘ 

ν—ˆλΈŒμ˜ λͺ¨λΈ 경둜 λ˜λŠ” λͺ¨λΈ 이름을 [`StableDiffusionPipeline`]에 μ „λ‹¬ν•˜μ—¬ 좔둠을 μœ„ν•΄ 파인 νŠœλ‹λœ λͺ¨λΈμ„ 뢈러올 수 μžˆμŠ΅λ‹ˆλ‹€:

<frameworkcontent>
<pt>
```python
from diffusers import StableDiffusionPipeline

model_path = "path_to_saved_model"
pipe = StableDiffusionPipeline.from_pretrained(model_path, torch_dtype=torch.float16)
pipe.to("cuda")

image = pipe(prompt="yoda").images[0]
image.save("yoda-naruto.png")
```
</pt>
<jax>
```python
import jax
import numpy as np
from flax.jax_utils import replicate
from flax.training.common_utils import shard
from diffusers import FlaxStableDiffusionPipeline

model_path = "path_to_saved_model"
pipe, params = FlaxStableDiffusionPipeline.from_pretrained(model_path, dtype=jax.numpy.bfloat16)

prompt = "yoda naruto"
prng_seed = jax.random.PRNGKey(0)
num_inference_steps = 50

num_samples = jax.device_count()
prompt = num_samples * [prompt]
prompt_ids = pipeline.prepare_inputs(prompt)

# shard inputs and rng
params = replicate(params)
prng_seed = jax.random.split(prng_seed, jax.device_count())
prompt_ids = shard(prompt_ids)

images = pipeline(prompt_ids, params, prng_seed, num_inference_steps, jit=True).images
images = pipeline.numpy_to_pil(np.asarray(images.reshape((num_samples,) + images.shape[-3:])))
image.save("yoda-naruto.png")
```
</jax>
</frameworkcontent>