Block loading behaviour in Drupal 8 in a performance sense through the visibility settings
The block is loaded (otherwise we could not check if he should be shown or not), but we do not call build() unless the block is visible and the user has access.

I'm pretty sure that was already the case in Drupal 7, it is only Drupal 6 AFAIK where it was building the blocks and then checking visiblity.