What is a 'hidden state' in BERT output?
BERT is a transformer.

A transformer is made of several similar layers, stacked on top of each others.
Each layer have an input and an output. So the output of the layer n-1 is the input of the layer n. 

The hidden state you mention is simply the output of each layer.



You might want to quickly look into this explanation of the Transformer architecture : 
https://jalammar.github.io/illustrated-transformer/

Note that BERT use only Encoders, no Decoders.