Synchronization of remote files download
If you change _retrieveFile to return a Task that's already started, you could simplify the whole Get() into just:

return _progress.GetOrAdd(key, _retrieveFile).Result;


Also, it seems that Downloader with empty map doesn't make sense, so you should check for that in your constructor, instead of using ??.