What is the proper train data format in LSTM?
[sentence_len, embedding_size] to [sentence_len, step_len, embedding_size] (step_len means present a word with it surround words).


Use step_len=1, just the current word. after squeezing the tensor, its shape will be  [batch_size,sentence_len,embedding_size].


  Previous sentence's last word may be affected by latter sentence's first few words.


Use padding, append the last 3 to 5 words of previous sentences at the start of the current sentences, do not backpropagate for the errors of those padded words, by masking their loss. as for the case of first sentences of paragraphs, padded with a simple &lt;pad&gt; token.


  doesn't need BiLSTM anymore? I think it still helps here.


Using BiLSTM is a must. if you want to limit step_len to be 1, by the way, you are computing the conditional probability of Y being true (correct/not correct) conditioned on the whole context (sentence). as you are feeding the sentence at once, the loss will be the summation over all single losses. 

Side note:
To batch the training examples, you also need to pad the end of sentences with the three to five words of the next ones. use &lt;pad&gt; for sentences ending the paragraph, don't permit their contributions to the model (adjusting the weight), mask your padded examples to have zero loss. because their labels are just noise.