How to input embeddings directly to a huggingface model instead of tokens?
I'm going over the huggingface tutorial where they showed how tokens can be fed into a model to generate hidden representations:
import torch
from transformers import RobertaTokenizer
from transformers import RobertaModel

checkpoint = 'roberta-base'
tokenizer = RobertaTokenizer.from_pretrained(checkpoint)
model = RobertaModel.from_pretrained(checkpoint)

sequences = [&quot;I've been waiting for a HuggingFace course my whole life.&quot;]

tokens = tokenizer(sequences, padding=True)
out = model(torch.tensor(tokens['input_ids']))
out.last_hidden_state

But how can I input word embeddings directly instead of tokens? That is, I have another model that generates word embeddings and I need to feed those into the model

Most (every?) huggingface encoder model supports that with the parameter inputs_embeds:
import torch
from transformers import RobertaModel

m = RobertaModel.from_pretrained(&quot;roberta-base&quot;)

my_input = torch.rand(2,5,768)

outputs = m(inputs_embeds=my_input)

P.S.: Don't forget the attention mask in case this is required.
