title: Stream uploading file to S3 on Node.js using formidable and (knox or aws-sdk)
body: <p>I'm trying to stream upload a file submitted via a form directly to an Amazon S3 bucket, using <a href="https://github.com/aws/aws-sdk-js" rel="noreferrer">aws-sdk</a> or <a href="https://github.com/LearnBoost/knox" rel="noreferrer">knox</a>. Form handling is done with <a href="https://github.com/felixge/node-formidable" rel="noreferrer">formidable</a>.</p>
  
  <p>My question is: how do I properly use formidable with aws-sdk (or knox) using each of these libraries' latest features for handling streams?</p>
  
  <p>I'm aware that this topic has already been asked here in different flavors, ie:</p>
  
  <ul>
  <li><a href="https://stackoverflow.com/questions/7505871/how-to-receive-an-uploaded-file-using-node-js-formidable-library-and-save-it-to">How to receive an uploaded file using node.js formidable library and save it to Amazon S3 using knox?</a></li>
  <li><a href="https://stackoverflow.com/questions/12526673/node-application-stream-file-upload-directly-to-amazon-s3">node application stream file upload directly to amazon s3</a></li>
  <li><a href="https://stackoverflow.com/questions/7860449/accessing-the-raw-file-stream-from-a-node-formidable-file-upload">Accessing the raw file stream from a node-formidable file upload</a> (and its very useful accepted answer on overiding form.onPart())</li>
  </ul>
  
  <p>However, I believe the answers are a bit outdated and/or off topic (ie. CORS support, which I don't wish to use for now for various reasons) and/or, most importantly, make no reference to the latest features from either aws-sdk (see: <a href="https://github.com/aws/aws-sdk-js/issues/13#issuecomment-16085442" rel="noreferrer">https://github.com/aws/aws-sdk-js/issues/13#issuecomment-16085442</a>) or knox (notably putStream() or its readableStream.pipe(req) variant, <a href="https://github.com/LearnBoost/knox/blob/master/Readme.md" rel="noreferrer">both explained in the doc</a>).</p>
  
  <p>After hours of struggling, I came to the conclusion that I needed some help (disclaimer: I'm quite a newbie with streams).</p>
  
  <p>HTML form:</p>
  
  <pre><code>&lt;form action="/uploadPicture" method="post" enctype="multipart/form-data"&gt;
    &lt;input name="picture" type="file" accept="image/*"&gt;
    &lt;input type="submit"&gt;
  &lt;/form&gt;
  </code></pre>
  
  <p>Express bodyParser middleware is configured this way:</p>
  
  <pre><code>app.use(express.bodyParser({defer: true}))
  </code></pre>
  
  <p>POST request handler:</p>
  
  <pre><code>uploadPicture = (req, res, next) -&gt;
    form = new formidable.IncomingForm()
    form.parse(req)
  
    form.onPart = (part) -&gt;
      if not part.filename
        # Let formidable handle all non-file parts (fields)
        form.handlePart(part)
      else
        handlePart(part, form.bytesExpected)
  
    handlePart = (part, fileSize) -&gt;
      # aws-sdk version
      params =
        Bucket: "mybucket"
        Key: part.filename
        ContentLength: fileSize
        Body: part # passing stream object as body parameter
  
      awsS3client.putObject(params, (err, data) -&gt;
        if err
          console.log err
        else
          console.log data
      )
  </code></pre>
  
  <p>However, I'm getting the following error:</p>
  
  <blockquote>
    <p>{ [RequestTimeout: Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.]</p>
    
    <p>message: 'Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.',
      code: 'RequestTimeout',
      name: 'RequestTimeout',
      statusCode: 400,
      retryable: false }</p>
  </blockquote>
  
  <p>A knox version of handlePart() function tailored this way also miserably fails:</p>
  
  <pre><code>handlePart = (part, fileSize) -&gt;
    headers =
      "Content-Length": fileSize
      "Content-Type": part.mime
    knoxS3client.putStream(part, part.filename, headers, (err, res) -&gt;
      if err
        console.log err
      else
        console.log res
    )      
  </code></pre>
  
  <p>I also get a big res object with a 400 statusCode somewhere.</p>
  
  <p>Region is configured to <strong>eu-west-1</strong> in both case.</p>
  
  <p>Additional notes:</p>
  
  <blockquote>
    <p>node 0.10.12</p>
    
    <p>latest formidable from npm (1.0.14)</p>
    
    <p>latest aws-sdk from npm (1.3.1)</p>
    
    <p>latest knox from npm (0.8.3)</p>
  </blockquote>