BertPunc (punctuation restoration with BERT) I've found the script for punctuation restoration. And I have one question about this method.
I will briefly explain the logic of the author. One of four tokens is assigned for each word: Other (0), PERIOD (1), COMMA (2), QUESTION (3). Further, all words are converted to BERT tokens. Here is an example:
  2045  0
  2003  0
  2200  0
  2210  0
  3983  0
  2301  0
  2974  0
  1999  0
  2068  2

Next, we do padding. We set a segment (e.g. eight words) and for each word we take two words before a token and four words after a token. Also, we add a padding token after each word. For the very first word, there are no words before. Therefore, a word are taken from the end. Similarly, for the last word, there are no words after and therefore a word are taken from the beginning.
Here is an example of it.
[1999, 2068, 2045, 0, 2003, 2200, 2210, 3983] 0
[2068, 2045, 2003, 0, 2200, 2210, 3983, 2301] 0
[2045, 2003, 2200, 0, 2210, 3983, 2301, 2974] 0
[2003, 2200, 2210, 0, 3983, 2301, 2974, 1999] 0
[2200, 2210, 3983, 0, 2301, 2974, 1999, 2068] 0
[2210, 3983, 2301, 0, 2974, 1999, 2068, 2045] 0
[3983, 2301, 2974, 0, 1999, 2068, 2045, 2003] 0
[2301, 2974, 1999, 0, 2068, 2045, 2003, 2200] 0
[2974, 1999, 2068, 0, 2045, 2003, 2200, 2210] 2

The first column contains tokens, and the second column contains punctuation symbols. In first column, &quot;0&quot; corresponds to a padding. Next we do TensorDataset, and than DataLoader. In the second column, '0' corresponds to &quot;other&quot;, and '2' corresponds to a &quot;period&quot;. Finaly we train a model:
for inputs, labels in data_loader_train:
       inputs, labels = inputs.cuda(), labels.cuda()
       output = model(inputs)

The algorithm works well, but I do not understand the following.
What's the point of putting padding in the middle? Maybe we can do punctuation restoration with BERT in a more simple way?
I am also confused by the method of BertPunc. Actually I can't believe it really works. Beacause the method use BertLanguageModel to do the job, and LanguseModel's main capabiltiy is to predict the next word of a word. So also use your example,  BertPunc inputs 8 words, the output of bert should be 8 next words of the inputs. But BertPunc added a Linear layer to only get 1 output value, which is to predict punctuation.

But I think we should use bert the way like the graph below: