Filter out duplicate entries in array?
Holy moly! I've solved it already, Batman. Check the array before adding the event to it, like so:

{% if event not in categoryEvents %}
        {% set categoryEvents = categoryEvents | merge([event]) %}
{% endif %}