How to alter response headers from module hook
in D7 this would happen with drupal_add_http_header()


In other hooks that might have been true, but hook_file_download works the same in Drupal 8 as it did in Drupal 7, without using drupal_add_http_header.

This is the documentation for both versions’ return value from that hook:


  If the user does not have permission to access the file, return -1. If the user has permission, return an array with the appropriate headers. If the file is not controlled by the current module, the return value should be NULL.


Emphasis mine.

You can't get access to the response object in hook_file_download, because it doesn't exist yet. It's not created until after the hook is invoked in FileDownloadController::download(). The headers you return are added to the response, so returning a suitable key-value array from the hook is the way to achieve what you want.