Targeted duplicate removal
Husk, 5 4 bytes
üoEė

Try it online!

Saved 1 byte thanks to Razetime
Saved 2 bytes thanks to Jo King

Thanks to Razetime for suggesting ü and Jo King for letting me know I could leave the superscript arguments out, saving me 2 bytes. It removes duplicates with a custom predicate that makes sure both arguments are equal to the number to be removed.
Explanation:
üoEė
ü     Remove duplicates by binary function (implicit second argument)
 o    Compose 2 functions
   ė  Make a list of 3 elements (first element is implicitly added)
  E   Are they all equal?