NLP varying amount of features and BoW as feature concatenating to feedforward NN I was looking at Google's Smart linkify machine learning models, as it closely relates to a personal project. And couldn't quite understand how the features are fed to the neural network.

It's about the following:




  Given a candidate entity span, we extract: Left context: five words
  before the entity, Entity start: first three words of the entity,
  Entity end: last three words of the entity (they can be duplicated
  with the previous feature if they overlap, or padded if there are not
  that many), Right context: five words after the entity, Entity
  content: bag of words inside the entity and Entity length: size of the
  entity in number of words. They are then concatenated together and fed
  as an input to the neural network.


I have 4 primary questions which I cant find a clear answer to:


The article specifies the features are concatenated. How does
a concatenation layer work internally? Does it concatenate all the
values in a single variable, in a very literal sense? how does that
work computationally?
How can a Bag of Words be a feature, when its a key-value pair? Or is it
also just all concatenated into one variable. Which again, how can
that work computationally?
The text specifies multiple words are used as a single feature; e.g. Left context: five words before the entity. Is this again concatenating the embedding / vectors?
Entity end: last three words of the entity (they can be duplicated with the previous feature if they overlap, or padded if there are not that many) does this mean a variable amount of features as input to the NN (or concatenation layer) or is this more intended as a configuration? Less context available so fewer amount of 'hard' coded input features? 


Perhaps a simple keras model with some hardcoded input variables would help shape the answer.

Is there more material online to understand and recreate the entity recognition model?

EDIT:

The article also mentions the lack of available context or entities. E.g. the diagram shows 4 features for entity (2 for entity start, 2 for entity end). The article mentions duplication, but duplicating 3 times doesnt sound like a great idea. Would a convolution layer with a filter (1x3) work better? 

And how would a Keras model look like then? Would it have two separate input layers? One input layer with 10 input features for the context + 1 feature for BoW + 1 feature for entity length. And another input layer with 4 input features followed by a convolution layer. And then both layers lead to a concatenate layer?
Let me take a crack at your questions:

The article specifies the features are concatenated. How does a concatenation layer work internally? Does it concatenate all the values in a single variable, in a very literal sense? how does that work computationally?


The concantenation of information in this context is, concatenation of vector represententations of the text. You could concatenate using a concatenation layer as described here. This is a very common approach followed where you want to feed your network information by taking various contexts depending on the problem you need to solve.

How can a Bag of Words be a feature, when its a key-value pair? Or is it also just all concatenated into one variable. Which again, how can that work computationally?


Bag of words, is typically a vector representation of the context. The above answer should help you. You could take a look at how to combine embeddings here as well

The text specifies multiple words are used as a single feature; e.g. Left context: five words before the entity. Is this again concatenating the embedding / vectors?


Yes, you are spot on. You combine the embedding vectors that you generate using skipgram or cbow, or you could even use one-hot encoded vectors.

Entity end: last three words of the entity (they can be duplicated with the previous feature if they overlap, or padded if there are not that many) does this mean a variable amount of features as input to the NN (or concatenation layer) or is this more intended as a configuration? Less context available so fewer amount of 'hard' coded input features?


Its always good to pad and use well defined dimensional vectors. Helps you structure your architecture better. You could use masking layer to ensure that the network ignores the paddings.

I hope this helps.