Omitting repeated content in a for loop
Currently you are trying to compare objects with each other and of course they are always different since they have a different reference. entry.vendor is not a single selected value but the field with all it's options and possible values.

Please try 

{% for entry in craft.entries.section('orderBook').order('vendor').all() %}
    {% if entry.vendor.value not in vendorArray %}
        {% set vendorArray = vendorArray|merge([entry.vendor.value]) %}
    {% endif %}
{% endfor %}