Concatenating embedding and hand-designed features for logistic regression
I think that I may understand where they are coming from. The key difference is if we're talking about inference or training.

Inference

There is nothing stopping you from concatenating your features. Whether they are hand-designed or a neural network's output, features can be combined however you please.

Training

If you don't intend to update your embeddings through training, then, there is no problem. Your embeddings will be treated as some other features and you can train your downstream model.
It gets tricky if you want to train both your downstream model and your embedding layer. Indeed, you will only be able to train your embedding layer if your full model (NN + downstream model) is differentiable. This is for instance not possible if the downstream model is a KNN model.
I assume this is what they meant when they said you couldn't concate embeddings and hand-designed features.