.

README.md

@@ -111,15 +111,17 @@ We segment videos using our ReLER lab's [SAM-Track](https://github.com/z-x-yang/
 ## 🔥 VideoGrain Editing
 
 ### Inference
+
 **prepare config**
+
 VideoGrain is a training-free framework. To run VideoGrain, please prepare your config follow these steps:
-- 1. Replace your pretrained model path and controlnet path in your config. you can change the control_type to `dwpose` or `depth_zoe` or `depth` (midas).
-- 2. Prepare your video frames and layout masks (edit regions) using SAM-Track or SAM2 in dataset config.
-- 3. Change the `prompt`, and extract each `local prompt` in the editing prompts. the local prompt order should be same as layout masks order.
-- 4. Your can change flatten resolution with 1->64, 2->16, 4->8. (commonly, flatten at 64 worked best)
-- 5. To ensure temporal consistency, you can set `use_pnp: True` and `inject_step:5-10`. (Note that pnp>10 steps will be bad for multi-regions editing)
-- 6. If you want to visualize the cross attn weight, set `vis_cross_attn: True`
-- 7. If you want to cluster DDIM Inversion spatial temporal video feature, set `cluster_inversion_feature: True`
+1. Replace your pretrained model path and controlnet path in your config. you can change the control_type to `dwpose` or `depth_zoe` or `depth` (midas).
+2. Prepare your video frames and layout masks (edit regions) using SAM-Track or SAM2 in dataset config.
+3. Change the `prompt`, and extract each `local prompt` in the editing prompts. the local prompt order should be same as layout masks order.
+4. Your can change flatten resolution with 1->64, 2->16, 4->8. (commonly, flatten at 64 worked best)
+5. To ensure temporal consistency, you can set `use_pnp: True` and `inject_step:5-10`. (Note that pnp>10 steps will be bad for multi-regions editing)
+6. If you want to visualize the cross attn weight, set `vis_cross_attn: True`
+7. If you want to cluster DDIM Inversion spatial temporal video feature, set `cluster_inversion_feature: True`
 
 ```bash
 bash test.sh 

config/demo_config.yaml

@@ -0,0 +1,48 @@
+pretrained_model_path: "./ckpt/stable-diffusion-v1-5"
+logdir: ./result/run_two_man/instance_level/3cls_spider_polar_vis_cross_attn
+
+dataset_config:
+    path: "data/run_two_man/run_two_man_fr2"
+    prompt: 'Man in red hoddie and man in gray shirt are jogging in forest'
+    n_sample_frame: 16
+    start_sample_frame: 0
+    sampling_rate: 2
+    layout_mask_dir: "./data/run_two_man/layout_masks_fr2"
+    layout_mask_order: ['left_man_plus','right_man_plus','trees','trunk']
+    negative_promot: "ugly, blurry, low res, unrealistic, unaesthetic"
+
+control_config:
+    control_type: "dwpose"
+    pretrained_controlnet_path: "./ckpt/control_v11p_sd15_openpose" 
+    controlnet_conditioning_scale: 1.0
+    hand: True
+    face: False
+
+editing_config:
+    use_invertion_latents: true
+    inject_step: 0
+    old_qk: 1
+    flatten_res: [1]
+    guidance_scale: 7.5 
+    use_pnp: false
+    use_freeu: false
+    editing_prompts: [
+            ['Spiderman and Polar Bear are jogging under cherry trees','man','Polar Bear','cherry trees',''],
+    ]
+        
+    clip_length: "${..dataset_config.n_sample_frame}"
+    sample_seeds: [0]
+    num_inference_steps: 50
+    blending_percentage: 0
+    vis_cross_attn: True
+    #cluster_inversion_feature: True
+    
+
+test_pipeline_config:
+    target: video_diffusion.pipelines.ddim_spatial_temporal.DDIMSpatioTemporalStableDiffusionPipeline
+    num_inference_steps: "${..validation_sample_logger.num_inference_steps}"
+
+
+
+seed: 42
+