Nested if-else within a for-loop using np.concatenate to create batches for a neural network
Your code is coupling independent ideas, this is making your code at least twice as long as you need it to be, as you have to duplicate your if statements.
And so I refactored your code in the following way:


Move x.append(np.concatenate(...)) out of the if statements. This is as you're needlessly duplicating this across them all.
Make the array passed to (1) outside the ifs, this is as you always add token_hidden and token_hidden_generator.
Replace once. generated_lemma = generated_lemma.replace('#', '|') and you don't have to call replace again.
Move all the generated_lemma code into a single if-else. Leading to:

generated_lemma = generated_lemma.replace('#', '|')
if generated_lemma in self.lemmas_word2vec_model:
    gen_lemma = self.lemmas_word2vec_model[generated_lemma]
else:
    gen_lemma = self.embedding_unknown_lemma

Cleanup the rest of the ifs.


This got me:

for generated_lemma, classified_lemma, token_hidden, token_hidden_generator in zip(
    tokens_generated,
    tokens_classified,
    token_dnn_output,
    token_generator_output
):
    generated_lemma = generated_lemma.replace('#', '|')
    if generated_lemma in self.lemmas_word2vec_model:
        gen_lemma = self.lemmas_word2vec_model[generated_lemma]
    else:
        gen_lemma = self.embedding_unknown_lemma

    classified_lemma = classified_lemma.replace('#', '|')
    if classified_lemma in self.lemmas_word2vec_model:
        class_lemma = self.lemmas_word2vec_model[classified_lemma]
    elif classified_lemma == '&lt;UNK&gt;':
        class_lemma = self.embedding_unknown_token
    else:
        class_lemma = self.embedding_unknown_lemma

    x.append(np.concatenate([
        gen_lemma,
        class_lemma,
        token_hidden,
        token_hidden_generator
    ]))


This may be faster, due to (3), however, I doubt it'd have any serious speed-ups. I don't really know any of the data types, and so can't comment on whether you need np.concatenate, or a faster way to add to x.