Concatenate two tensors of different shape from two different input modalities
I have two tensors:
a = torch.randn((1, 30, 1220)) # represents text embedding vector (30 spans, each with embedding size of 1220)
b = torch.randn((1, 128, 256)) # represents image features obtained from a pretrained CNN (object detection)


How do I concatenate everything in b to each one of the 30 spans of a?

How to concatenate the whole b to the whole a?


This is what I'm trying to do:

The authors have only provided this text:

I'm extracting features (outlined in red) from a 3d point cloud (similar to CNN but for 3d) as shown below:


Since these representations are from two different modalities (i.e., text and image) and they contain valuable features that are of great importance to the final goal, I would suggest to fuse them in a &quot;learnable&quot; manner instead of a mere concatenation or addition. Furthermore, such a learnable weighting (between features) would be optimal since in some cases one representation would be far more useful than the other whereas at other instances the vice versa applies.
Please note that a mere concatenation can also happen in this fusion module that you would implement. For the actual implementation, there are several types of fusion techniques. E.g. Simple Fusion, Cold Fusion etc. (cf. Fusion Models for Improved Visual Captioning, 2021)
For instance, one straightforward idea would be to use a simple linear layer to project one of the features to the same dimensionality as the other and then do a simple concatenation, with some optional non-linearity if needed.
