Implementing the Dependency Sensitive CNN (DSCNN ) in Keras
Intermediately I finished the network and it is working great :-) Anyone who is also looking for this network can download it on my Github page.
To refer directly to my questions:
Question 1:
The proposed code actually makes partially sense:

One should use the shared-embedding layer to save memory usage, so this part makes sense
In the initially proposed code I used TimeDistributed layer, this is the wrong way for the purpose of the network. We actually have to use a simple shared LSTM layer. Time distributed layers are needed for many-to-many networks. For more information about this, I refer to karpathy's blog and Brownlees blog.
I applied the second LSTM layer before the concatenation, with this I applied two LSTMs to capture the joint meaning of the sentences. That is not the intention of the DSCNN, so we have to apply the one LSTM layer after the concatenation

So far the code looks like:
sentence_inputs = [Input(shape=(max_sentence_len, embedding_dim,), name=&quot;input_&quot; + str(i))
                   for i in range(max_sentences_per_doc)]

# LSTMs and Average Pooling (sentence-level)
shared_sentence_lstm = LSTM(units=embedding_dim, return_sequences=True, activation='tanh')
shared_average_pooling = AveragePooling1D(pool_size=max_sentence_len)
sentence_modeling = [shared_sentence_lstm(sentence_inputs[i]) for i in range(max_sentences_per_doc)]
sentence_modeling = [shared_average_pooling(sentence_modeling[i]) for i in range(max_sentences_per_doc)]

doc_modeling = Concatenate(axis=1)(sentence_modeling)
doc_modeling = LSTM(units=embedding_dim, activation='tanh', return_sequences=True)(doc_modeling)

The convolutional layer with multiple filter sizes has to be applied &quot;manually&quot;, therefore we run the convolutions in a loop and concatenate the resulting layers after we flattened them (you could also apply GlobalMaxPooling, then you would not have to flatten the layer, but with this you would lose many information of your feature vectors).
conv_blocks = []
for sz in kernel_sizes:
    conv = Convolution1D(filters=filters,
                         kernel_size=sz,
                         padding=&quot;valid&quot;,
                         activation=&quot;relu&quot;,
                         strides=1)(doc_modeling)
    conv = MaxPooling1D(pool_size=2)(conv)
    conv = Flatten()(conv)
    conv_blocks.append(conv)
doc_modeling = Concatenate()(conv_blocks) if len(conv_blocks) &gt; 1 else conv_blocks[0]


After this we can normally apply an activation layer (e.g. softmax or sigmoid, with or without dropout) and compile the model

Question 2 and 3: We need either an embedding directly in the batch-data or the embedding layer.
If you implement your network to have the embedding already in the batch data, you have to generate it only once. It works with small datasets and small embedding sizes (small depends on your memory size etc.), but with large datasets (e.g. 2000 words per document and an embedding dimension of 300) you rapidly exceed like 30gb of memory. Therefore it definitely makes sense to use the embedding layer, since it stores each word vector only once. The downside with this embedding layer is, that it has to perform the word index to word vector replacement after each batch, this costs a little bit of runtime, but since it is only a lookup operation this is not very expensive. So be encouraged to use the embedding layer.
Question 4:
It would not. Firstly, because of some parameter issues (e.g. Conv1D accepts only one integer for the kernel_size), secondly, it implements another architecture, that makes no sense (cf. question 1, TimeDistributed layer).
Question 5: No :-), see question 4