                                                                                                                                                                                                                                                                                                                  
By far most of the memory is taken up by the cross-attention layers. Instead of running this operation in batch, one can run it sequentially 