# Best way to use a BERT transformer on each sentence in a document?

So, I’m not quite sure how to put this, but I’ll try my best. Also, the actual model is rather complex, so I’ll simplify it a bit. So, my goal is to predict the sentiment of each sentence in a document from the corpus. Now, my naïve approach was to treat each each sentence as independent from each other, i.e. the order doesn’t matter, thus each sentence can be treated as a separate input in the model. So, my model would look something like this:

class my_model(Bert_model):
def init(self, bert_base_model, args):
# some stuff
self.bert_for_global_context = bert_base_model
self.dense = torch.nn.Linear(self.embed_dim, 3)
self.pooler = BertPooler(config)
def forward(self, input):
# more stuff
global_context_out = self.bert_for_global_context(input)[‘last_hidden_state’]
pooled_output = self.pooler(global_context_out)
logits = self.dense(pooled_output)
loss_fun = CrossEntropyLoss()
loss = loss_fun(logits, labels)
return loss
model = my_model(bert_base_model, args)

# training

input = batch
loss = model(input)

# more stuff

Note that the input is of shape [batch size, sentence length]. So, I was thinking about how to model any dependency between the sentences, so I was thinking about modeling the sequence of sentiments using a conditional random field, using this package: pytorch-crf — pytorch-crf 0.7.2 documentation. To do this, I reshaped my dataset so that each input would be of size [batch size, document length, sentence length] . So, my model now looks like this:

class my_model(Bert_model):
def init(self, bert_base_model, args):
# some stuff
self.bert_for_global_context = bert_base_model
self.dense = torch.nn.Linear(self.embed_dim, 3)
self.pooler = BertPooler(config)
self.crf = CRF(3, batch_first=True)
def forward(self, input):
# more stuff
global_context_out = self.bert_for_global_context(input)[‘last_hidden_state’]
pooled_output = self.pooler(global_context_out)
logits = self.dense(pooled_output)
logits = logits.view(batch size, document length, 3)
loss_fun = self.crf
loss = loss_fun(logits, labels)
return loss
model = my_model(bert_base_model, args)
{"ft_lang_label":"__label__en","ft_lang_prob":0.7777117,"math_prob":0.9454265,"size":2685,"snap":"2022-27-2022-33","text_gpt3_token_len":658,"char_repetition_ratio":0.12010444,"word_repetition_ratio":0.23466666,"special_character_ratio":0.23947859,"punctuation_ratio":0.17892644,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":0.99557585,"pos_list":[0],"im_url_duplicate_count":[null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2022-07-02T09:12:56Z\",\"WARC-Record-ID\":\"<urn:uuid:79d0bc5a-0a31-4fe2-8262-386666c8875d>\",\"Content-Length\":\"18825\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:6dbad245-ef1c-4282-9726-e1574b5370c1>\",\"WARC-Concurrent-To\":\"<urn:uuid:12c791c9-5928-4b96-8016-655e34392c08>\",\"WARC-IP-Address\":\"64.62.250.111\",\"WARC-Target-URI\":\"https://discuss.huggingface.co/t/best-way-to-use-a-bert-transformer-on-each-sentence-in-a-document/8290\",\"WARC-Payload-Digest\":\"sha1:7TZDZRW5ZUZFF6B6WPKBPCH5GP24X4AI\",\"WARC-Block-Digest\":\"sha1:VWVEZSODL4RDKYTEFF5BYZ3VSQKJBS2M\",\"WARC-Identified-Payload-Type\":\"text/html\",\"warc_filename\":\"/cc_download/warc_2022/CC-MAIN-2022-27/CC-MAIN-2022-27_segments_1656103989282.58_warc_CC-MAIN-20220702071223-20220702101223-00226.warc.gz\"}"}