How do I unset a block?
Yes, what you are seeing is the render cache. If you are developing a theme, you likely want to disable render caching and enable twig debug. See sites/example.settings.local.php for instructions on how to do that.

That said, breadcrumbs, like almost everything else, is a block now. Blocks are configuration. If you don't want a block, just remove that block in the UI. If you just remove it in the template, Drupal still needs to load and prepare that block for rendering as it cannot know the template doesn't print it.

And if you don't display the region (you always display regions with 0-N blocks, not directly blocks in page template), you should also not define it as an available region so that the user/site admin knows that he can't place blocks there.