title: Is it dangerous to pipe data from S3 to a long-running process
body: <p>I have a 30GB file and I want to feed it into a program that accepts data via stdin and that will take 24hr to process the data.</p>
  <p>Can I just do <code>aws s3 cp s3://bigfile.txt - | long_process.sh</code>?</p>
  <p>I'm attracted to this b/c I don't have to store bigfile.txt on disk and I can start working on the data stream immediately, but I worry that if the s3 command has a problem during the 24 hours that it will crash and I will lose all the progress.</p>
  <p>EDIT:</p>
  <p>Alternatively, I am looking for a way to stage the data to a file and read from the staged data.  For example: <code>aws s3 cp s3://bigfile.txt /local/bigfile.txt &amp;; long_process.sh &lt; /local/bigfile.txt</code>.  The trouble is that long_process.sh complains of a truncated file; It does not seem to wait for further input like a pipe would.  I have thought about <code>tee</code> and <code>mkfifo</code> but nothing seems to quite fit my needs.</p>