Why does the BERT NSP head linear layer have two outputs?
Here's the code in question. 

https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L491

class BertOnlyNSPHead(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.seq_relationship = nn.Linear(config.hidden_size, 2)

    def forward(self, pooled_output):
        seq_relationship_score = self.seq_relationship(pooled_output)
        return seq_relationship_score


I think it was just ranking how likely one sentence would follow another? Wouldn't it be one score?

The two scores are meant to represent unnormalized probabilities (logits) from the model. If we softmax them, we get our predictions, where index 0 indicates next sentence, and index 1 indicates random.

This is just a stylistic choice on the HuggingFace author's behalf, probably to keep the loss function consistent.

Here's the forward method of BertForPretraining, where self.cls is BertOnlyNSPHead:

    prediction_scores, seq_relationship_score = self.cls(sequence_output, pooled_output)

    outputs = (prediction_scores, seq_relationship_score,) + outputs[
        2:
    ]  # add hidden states and attention if they are here

    if masked_lm_labels is not None and next_sentence_label is not None:
        loss_fct = CrossEntropyLoss()
        masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), masked_lm_labels.view(-1))
        next_sentence_loss = loss_fct(seq_relationship_score.view(-1, 2), next_sentence_label.view(-1))
        total_loss = masked_lm_loss + next_sentence_loss
        outputs = (total_loss,) + outputs


it's convenient to use the same CrossEntropyLoss for both MLM and NSP.

As you describe, it would be equivalent to have NSP produce a single output, then feed that number through a sigmoid to get probability of the next sentence. We can then train with BCEWithLogitsLoss. (where BCE is just the special binary case of cross entropy loss).
