Constraint on multiple columns?
A UNIQUE, multicolumn, partial expression index should do the trick:

CREATE UNIQUE INDEX ON edges (LEAST(src, tgt), GREATEST(src, tgt), rel)
WHERE rel_type = 2;


Works with INSERT ... ON CONFLICT DO NOTHING.

Related:


Unique postgres constraint for permutations across multiple columns
How to create a postgres table with unique combined primary key?
PostgreSQL multi-column unique constraint and NULL values


Aside: If those are IP addresses, consider the data type cidr or ip4 from the additional module ip4r for column rel.