How to set order for constraints to be checked in Postgres
It's hard to imagine a use case where it matters. Maybe a case where one constraint is much more likely to fail and/or the other one(s) are expensive (huge tables?) and you'd rather check the one first to optimize performance? Be that as it may, to answer the question asked:
CHECK constraints are applied alphabetically. The manual:

When a table has multiple CHECK constraints, they will be tested for each row in alphabetical order by name, after checking NOT NULL constraints.

But there is no such rule for UNIQUE constraints.
Slow and sure
A slow and sure way: define competing constraints as DEFERRABLE. The manual:

A constraint that is not deferrable will be checked immediately after every command. Checking of constraints that are deferrable can be postponed until the end of the transaction ...

But if the purpose of this is to optimize performance, this may defy the purpose. The manual warns:

Be aware that this can be significantly slower than immediate uniqueness checking.

There are also a couple of side effects. Like: FK constraints cannot currently point columns with only a deferrable UNIQUE constraint. Read the manual carefully. Related:

Constraint defined DEFERRABLE INITIALLY IMMEDIATE is still DEFERRED?

Quick and dirty
Some quick tests indicate that UNIQUE constraints are checked in the order of their creation. Could be bound to the oid in pg_constraint or the oid of the underlying unique index ... But neither can I give any guarantees that this is true, nor does the Postgres project guarantee anything in that direction. I found nothing in the manual or anywhere. Looks like an implementation detail that's not reliable. But for the sole purpose of performance optimization &quot;not reliable&quot; may still be good enough.
To take advantage of this observation create instance_source_uniq before you create instance_name_uniq. (DROP and recreate other UNIQUE constraints that were created earlier.) Check success with a forged INSERT that violates all competing UNIQUE constraints at once. You'll see an error message for the one that's checked first.
To further keep that order intact across dump / restore cycles, rename instance_source_uniq to something like instance_1st_source_uniq, which sorts before all other competing constraint names. pg_dump dumps objects in alphabetical order, so this should result in your favored constraint being created first on restore.

Postgres pg_dump dumps database in a different order every time (Peter Eisentraut is a core developer.)

Again, no guarantees. Whenever instance_1st_source_uniq is dropped and recreated, you may lose the effect. Or if Postgres decides to do things differently. Or if there are exception to the observed behavior.