You can adjust the merging ratio with a parameter called α (alpha) in the paper, or scale in our implementation. You can tweak it with the following code, that passes scale as cross_attention_kwargs in the pipeline call: