$form_state->setErrorByName() for a form in a block doesn't work properly
This is a known bug. You can see the same problem with the core user login block upon submitting bad credentials. The comments in that issue note that it's a side effect of another issue which has a fix committed to the 8.2.x branch.