title: Moving metadata storage for cluster jobs with snakemake
body: <p>I am using snakemake to submit jobs from an AFS filesystem to a HTCondor queue. The submission starts off fine. Jobs submit and complete successfully. But after a certain point, submission crashes with the following error:</p>
  
  <pre><code>OSError: [Errno 27] File too large: '/afs/cern.ch/work/a/avenkate/private/MyKstTauTau/.snakemake/metadata/cm9vdDovL2Vvc2xoY2IuY2Vybi5jaC8vZW9zL2xoY2IvdXNlci9hL2F2ZW5rYXRlL0tzdFRhdVRhdS9Cc0JrZ01DL1ByZXNlbC8yMDE2X01VLzIwMjBfMDQvMzY1OTMyLzM2NTkzMjE4Ny9udHVwbGVfVE0ucm9vdA=='
  </code></pre>
  
  <p>When I look up this error, I understand from <a href="https://sites.google.com/a/stanford.edu/rcpedia/how-do-i/why-do-i-keep-getting-errno-27-file-too-large" rel="nofollow noreferrer">https://sites.google.com/a/stanford.edu/rcpedia/how-do-i/why-do-i-keep-getting-errno-27-file-too-large</a> that it has to do with AFS, and not snakemake. It seems to be a limit on the number of files in a directory in AFS (The error message itself is misleading. This file size of the metadata files is in bytes. But the filenames are long).</p>
  
  <p>So I would like to know if there is any way of getting around this issue with metadata files piling up. I have 10268 files in my metadata directory when the crash happens, and all are from one snakemake run. </p>
  
  <p>Is it possible to have snakemake not store the metadata files locally, but rather in the temp directory of whatever node the job runs on? I see in the command line options that we can specify <code>--shadow-prefix</code> to tell snakemake where to store the shadow directory. Is there a similar functionality for the metadata? </p>
  
  <p>I would appreciate any input on this. My submit command is </p>
  
  <pre><code>snakemake     --cluster 'condor_qsub -V -l procs={threads}' --forceall    --latency-wait 30     --rerun-incomplete     --max-jobs-per-second 100     --printshellcmds     --restart-times 0     --local-cores 8     --keep-going     -j 100     -s Snakefile_Data   --shadow-prefix $TMPDIR/MyKstTauTau.snakemake```
  </code></pre>