Update README.md

README.md

@@ -158,3 +158,48 @@ image
 
 
 ![image/png](https://cdn-uploads.huggingface.co/production/uploads/623ce1c6b66fedf374859fe7/qFBQG-smW5W75MBaNwZcH.png)
+
+## Fine-tuning script 
+
+Download this script: [SD3 DreamBooth-LoRA_Fine-Tune.ipynb](https://huggingface.co/lamm-mit/stable-diffusion-3-medium-leaf-inspired/resolve/main/SD3_DreamBooth-LoRA_Fine-Tune.ipynb)
+
+You need to create a local folder ```leaf_concept_dir_SD3_12``` and add the leaf images (provided in this repository, see subfolder). The code will automatically download the training script. The training script can handle custom prompts associated with each image, which are generated using BLIP.
+
+For instance, for the images used here, they are:
+
+```raw
+['<leaf microstructure>, a close up of a green plant with a lot of small holes',
+ '<leaf microstructure>, a close up of a leaf with a small insect on it',
+ '<leaf microstructure>, a close up of a plant with a lot of green leaves',
+ '<leaf microstructure>, a close up of a green plant with a yellow light',
+ '<leaf microstructure>, a close up of a green plant with a white center',
+ '<leaf microstructure>, arafed leaf with a white line on the center',
+ '<leaf microstructure>, a close up of a leaf with a yellow light shining through it',
+ '<leaf microstructure>, arafed image of a green plant with a yellow cross']
+```
+
+The Parquet dataset generated during pre-calculation of embeddings is stored in the folder ```{data_df_path}```. It includes the image paths, embeddings, and a few other columns that are used by the training script. 
+
+Training then proceeds as:
+
+```raw
+accelerate launch train_dreambooth_lora_sd3_miniature.py \
+  --pretrained_model_name_or_path="{pretrained_model_name_or_path}" \
+  --instance_data_dir="{instance_data_dir}" \
+  --data_df_path="{instance_output_dir_embed}" \
+  --output_dir="{instance_output_dir}" \
+  --mixed_precision="fp16" \
+  --instance_prompt="{instance_prompt}" \
+  --resolution=1024 \
+  --train_batch_size=1 \
+  --gradient_accumulation_steps=4 \
+  --gradient_checkpointing \
+  --learning_rate=1e-4 \
+  --lr_scheduler="constant" \
+  --weighting_scheme="logit_normal" \
+  --lr_warmup_steps=0 \
+  --use_8bit_adam \
+  --max_train_steps=500 \
+  --checkpointing_steps=500 \
+  --seed="3234290"
+```