Combining textual and numeric features into pre-trained Transformer BERT I have a dataset with 3 columns:

Text
Meta-data (intending to extract features from it, then use those i.e., numerical features)
Target label

Question 1: How can I use a pre-trained BERT instance on more than the text?
One theoretical solution suggests having BERT fed the text and another neural network with the numerical features fed into this one, then aggregating their output, into another neural network.

Is that the most efficient approach?

Question 2: How can you connect neural networks?

You get the output from each, but then what?

You get classification output from BERT, you get classification
output from MLP based on numerical features.

You concatenate these and feed them to another MLP, and you get the
final prediction? Wouldn't that last prediction be less robust ?

In other words, does the last MLP encapsulate the other 2 networks?

If so, what happens if BERT predicts on 90%, but the first MLP just
50%, will we get a lesser outcome?


Question 3: Any tips on how to implement this in pytorch?
When they are talking about aggregating their outputs, they mean the final embeddings (just before the classification layer) not the output of the network itself.
You take the embeddings from both of the networks and concatenate them vertically. This concatenated embedding you use to predict the final output you wanted.
Torch has a function torch.cat for this purpose.