How do I use a term as contextual filter correctly?
You don't need Relationships for matching through ER fields, IDs are enough so it's enough to use the ER field Contextual matching directly.
Instead of the ID Contextual filter add the Contextual filter for the ER field and configure it:
When the filter value is NOT available
Provide default value
Taxonomy term ID from URL
Load default filter from node page, that's good for related taxonomy blocks
Limit terms by vocabulary
[Your vocabulary]
Filter to items that share any term
When the filter value IS available or a default is provided
Specify validation criteria
Taxonomy term ID
[Your vocabulary]
Set other options according to your needs.
You can test it by entering a Taxonomy ID into the Views Preview field, you should get only the nodes that reference that term.
When you place the Block in a theme region also configure the Contextual option, it won't work without it.