Show all nodes with the same term, the current user is assigned to That sounds quite simple but I can't find a hint how to do this:

I have invoices with a term reference to a "Department" vocabulary [field_invoice_department]. The user has a term reference to the same vocabulary [field_user_department].

So how should I configure my contextual filters to show all invoices assigned to the same department, the current user is assigned to?
There are a lot of answers you can search for "sibling taxonomy" or some other wording. 

Here's one answer you can follow: Filter taxonomy term attached to user field

The only difference is you always want your views to show the current user's matches, and not create, for example, profile pages for everyone separately as it is in that answer. 

To make your Views show only the current user's matches in the User ID Contextual filter you have to choose User ID from logged in user instead of User ID from route context.