Concatenate layers with different sizes in PyTorch
In Keras, it is possible to concatenate two layers of different sizes:
# Keras — this works, conceptually
layer_1 = Embedding(50, 5)(inputs)
layer_2 = Embedding(300, 20)(inputs)
concat = Concatenate()([layer_1, layer_2])
# -&gt; `concat` now has shape `(*, 25)`, as desired

But PyTorch keeps complaining that the two layers have different sizes:
# PyTorch — this does not work
class MyModel(torch.nn.Module):
    def __init__(self):
        self.layer1 = Embedding(50, 5)
        self.layer2 = Embedding(300, 20)

    def forward(self, inputs):
        layer_1 = self.layer1(inputs)
        layer_2 = self.layer2(inputs)
        concat = torch.cat([layer_1, layer_2])

The code just above results in this error:

RuntimeError: Sizes of tensors must match except in dimension 0.
Expected size 5 but got size 20 for tensor number 1 in the list.

The final concat layer I want is a layer of size 25 made of the concatenation of the two source layers.
As the two source layers are Embedding layers, I do not see as optimal that they would share the same dimension. In this example, using an embedding dimension of 5 for a vocabulary of 50 items, and an embedding dimension of size 20 for a vocabulary of 200 items.
How should this problem be solved in PyTorch?

Indeed torch.cat will apply the concatenation on the first axis. Since you are looking to concatenate on the second axis, you should provide the dim argument as:
&gt;&gt;&gt; concat = torch.cat([layer_1, layer_2], dim=1)

