Category query very slow on multilingual site
It's hard to know for sure without seeing your query profile output, but try eager loading your ports field like so:

{% set categories = craft.categories({
    group: 'locations',
    limit: null,
    with: ['ports']
}) %}