Set that keeps unique categories of objects I would like to know if this type of special set operator exists, and if yes what is it called and if it has any other special properties. 

Lets say I have this set $S$ of items. Like all sets, if the same item is added twice in the set, the set will not add it again. However, when adding an item, using this special operator, it also checks if it is 'better' than another item already in the set, and if yes it replaces it with the new item rather than letting both.

For a more concrete example:

$S_1 = \{apple, orange, pear\}$

$S_2 = \{betterApple, orange, banana\}$

$S = S_1 \cup_\succ S_2 = \{betterApple, orange, pear, banana\} $

So in this case $\cup_\succ$ checked some relationship, lets say $\succ$ where $betterApple \succ apple$, and if such a relationship existed it dropped $apple$ rather than allowing both in the set, keeping the stronger one in the set.
There is no official operation that accomplishes what you want, but you can always define an operation $\cup_{\succ}$ which does so. Just define the operation in words.



If for some reason you want a "physical" implementation, then here is one way. Each set is a collection of pairs, i.e.
$$
\begin{align*}
S_1 &amp;= \{ (apple,apple), (orange,orange), (pear,pear) \}, \\
S_2 &amp;= \{ (apple,betterApple), (orange,orange), (banana,banana) \}.
\end{align*}
$$
Furthermore, there is some partial order $\prec$ such that $(x,y),(z,w)$ are comparable iff $x = z$. When you take the union $S_1 \cup S_2$, you get
$$
S_1 \cup S_2 = \{ (apple,apple), (apple,betterApple), (orange,orange), (pear,pear), (banana,banana) \}.
$$
Then you can define a function
$$
\phi(S) = \{ x \in S : \text{ no $y \in S$ satisfies $y \succ x$ } \}.
$$
What you want is $\phi(S_1 \cup S_2)$.