Give access to private files to certain role?
I found the problem. As per this page:

Once configured, files stored in the private directory are inaccessible via a direct link; however, if Drupal constructs a link to the file, the file will be accessible to anyone who can see the link.
For example: you have created a new content type with a file field which stores files in your site's private file directory. Next you create a node from this new content type and attach two new files. When the node is published links to both attached files are visible and anyone who can view the node may download the files. Now, if you unpublish the node, all attached files become inaccessible for download even if you use the direct link to the files that worked when the node was published.
Re-publish the node, and disable the &quot;display&quot; checkbox for one of the files. Save the node. Now one file is accessible for public download and the other is not accessible--even if you have the direct URL for the file that is not listed you will not be able to download this file.

So the problem was that I was expecting you could access a file if you have the right URL, even if that file is used on an UNPUBLISHED node. So to solve my problem I:

Published all the nodes I needed to have access to the image
Used &quot;Rabbit Hole&quot; to hide the nodes from the public