title: Hadoop DistributedCache caching files without absolute path?
body: <p>I am in the process of migrating to YARN and it seems the behavior of the DistributedCache changed.</p>
  
  <p>Previously, I would add some files to the cache as follows:</p>
  
  <pre><code>for (String file : args) {
     Path path = new Path(cache_root, file);
     URI uri = new URI(path.toUri().toString());
     DistributedCache.addCacheFile(uri, conf);
  }
  </code></pre>
  
  <p>The path would typically look like </p>
  
  <pre><code>/some/path/to/my/file.txt
  </code></pre>
  
  <p>Which pre-exists on HDFS and would essentially end up in the DistributedCache as </p>
  
  <pre><code>/$DISTRO_CACHE/some/path/to/my/file.txt
  </code></pre>
  
  <p>I could symlink to it in my current working directory and use with <code>DistributedCache.getLocalCacheFiles()</code></p>
  
  <p>With YARN, it seems this file instead ends up in the cache as:</p>
  
  <pre><code>/$DISTRO_CACHE/file.txt
  </code></pre>
  
  <p>ie, the 'path' part of the file URI got dropped and only the filename remains. </p>
  
  <p>How does with work with different absolute paths ending up with the same filename? Consider the following case: </p>
  
  <pre><code>DistributedCache.addCacheFile("some/path/to/file.txt", conf);
  DistributedCache.addCacheFile("some/other/path/to/file.txt", conf);
  </code></pre>
  
  <p>Arguably someone could use fragments:</p>
  
  <pre><code>DistributedCache.addCacheFile("some/path/to/file.txt#file1", conf);
  DistributedCache.addCacheFile("some/other/path/to/file.txt#file2", conf);
  </code></pre>
  
  <p>But this seems unnecessarily harder to manage. Imagine the scenario where those are command-line arguments, you somehow need to manage that those 2 filenames, although different absolute paths would definitely clash in the DistributedCache and therefore need to re-map these filenames to fragments and propagate as such to the rest of the program?</p>
  
  <p>Is there an easier way to manage this? </p>