Craft 2 use different template for the same structure
I finally managed to answer my own question. I changed my routing to

/projects/slug/slug/gallery to projects/gallery/_entry

So was able to use a proper absolute link with /gallery added to the end

.com/projects/category/project/gallery

In the gallery _entry I used the following code to get the entry

{% set projectSlug = craft.request.getSegment(-2) %}
{% set categorySlug = craft.request.getSegment(-3) %} 
{% set category = craft.categories.slug(categorySlug).first() %}
{% set entry = craft.entries.section('projects').slug(projectSlug).relatedTo(category).first() %}

This then worked fine, even if the same named project was used in two different categories

I would still prefer to do this without routing if anyone has any suggestions please?