title: Lambda S3 getObject stream to ES ingest pipeline
body: <p>I'm using lambda to push PDF files uploaded to S3 over to AWS hosted elasticsearch with the ES ingest attachment plugin. </p>
  
  <p>Some of the PDFs can be large - over 100mb - and it seems like it would be best to stream the files in chunks rather than load the entire file buffer into lambda memory. The s3-lambda-es sample code I've seen all work on json logs which are easy to stream, but I haven't seen anybody working with binary documents for the ingest plugin. </p>
  
  <p>Note: I'm using the elasticssearch-js library with http-aws-es connectionClass to make it work with lambda. </p>
  
  <p>My current code basically works, but it loads the entire pdf before sending to ES. 
  Here is the getObject callback: </p>
  
  <pre><code>s3.getObject({ Bucket: bucket, Key: key }, function(err, data) {
      if (err) {
        console.log(err, err.stack); // an error occurred
        context.fail();
      } else {
        console.log('data: ', data); // successful response
        var attachment = data.Body.toString('base64');
        elasticsearch.index(
          {
            index: 'attachments2',
            pipeline: 'pdf_attachment2',
            type: 'pdf',
            body: {
              data: attachment
            }
          },
          function(error, resp) {
            if (error) {
              console.trace('index error!', error);
            } else {
              console.log('index response', resp);
              context.succeed();
            }
          }
        );
      }
    });
  </code></pre>
  
  <p>Is there a better way to stream a single large document to ES?</p>