How to use custom block view mode in View?
Don't use custom blocks, use nodes or a custom entity type. Custom blocks are designed to be displayed as a block, not a view inside a block. They don't have a template on their own. See https://www.drupal.org/node/2704331.

Yes, it's not great, but it is unlikely that this will change any time soon.