BERT - How Question answering is different than classification
For Question Answering, you need 2 logits : one for the start position, one for the end position. Based on these 2 logits, you have an answer span (denoted by the start/end position).
In the source code, you have : pooled_output = self.pooler(sequence_output)
If you take a look at the pooler, there is a comment :  


# We "pool" the model by simply taking the hidden state corresponding
# to the first token.


So the sequence output is all the token representations, while the pooled_output is just a linear layer applied to the first token of the sequence.


In classification case, you just need a global representation of your input, and predict the class from this representation. No need to access every token of the sequence, you want a single representation of the sequence.
But in the Question Answering case, in order to predict the answer span, you need to decide which token will be the start and which token will be the end of the span. For this, you need to consider every token independently.