title: Is it theoretically possible to use `putRecord` in Kinesis via singular http POST request?
body: <p>I've been experiencing some issues with AWS Kinesis inasmuch as I have a stream set up and I want to use a standard http <code>POST</code> request to invoke a Kinesis PutRecord call on my stream. I'm doing this because bundle-size of my resultant javascript application matters and I'd rather not import the <code>aws-sdk</code> to accomplish something that should (on paper) be possible.</p>
  
  <p>Just so you know, I've looked at <a href="https://stackoverflow.com/questions/39939225/call-rest-api-for-amazon-kinesis-with-setting-up-api-gateway">this other stack overflow question about the same thing</a> and It was... sort of informational.</p>
  
  <p>Now, I already have a method to sigv4 sign a request using an access key, secret token, and session token. but when I finally get the result of signing the request and send it using the in-browser <code>fetch</code> api, the service tanks with  (or with a json object citing the same thing, depending on my Content-Type header, I guess) as the result. </p>
  
  <p>Here's the code I'm working with </p>
  
  <pre><code>// There is a global function "sign" that does sigv4 signing
  // ...
  
  var payload = {
      Data: { task: "Get something working in kinesis" },
      PartitionKey: "1",
      StreamName: "MyKinesisStream"
  }
  
  var credentials =  {
      "accessKeyId": "&lt;access.key&gt;",
      "secretAccessKey": "&lt;secret.key&gt;",
      "sessionToken": "&lt;session.token&gt;",
      "expiration": 1528922673000
  }
  
  function signer({ url, method, data }) {
      // Wrapping with URL for piecemeal picking of parsed pieces
      const parsed = new URL(url);
  
      const [ service, region ] = parsed.host.split(".");
  
      const signed = sign({
          method,
          service,
          region,
          url,
  
          // Hardcoded
          headers : {
              Host           : parsed.host,
              "Content-Type" : "application/json; charset=UTF-8",
  
              "X-Amz-Target" : "Kinesis_20131202.PutRecord"
          },
  
          body : JSON.stringify(data),
      }, credentials);
  
      return signed;
  }
  
  // Specify method, url, data body
  var signed = signer({
      method: "POST",
      url: "https://kinesis.us-west-2.amazonaws.com",
      data : JSON.stringify(payload)
  });
  
  var request = fetch(signed.url, signed);
  </code></pre>
  
  <p>When I look at the result of request, I get this:</p>
  
  <pre><code>{
      Output: { 
         __type: "com.amazon.coral.service#InternalFailure"},
         Version: "1.0" 
  }
  </code></pre>
  
  <p>Now I'm unsure as to whether Kinesis is actually failing here, or if my input is malformed?</p>
  
  <p>here's what the signed request looks like</p>
  
  <pre><code>{
      "method": "POST",
      "service": "kinesis",
      "region": "us-west-2",
      "url": "https://kinesis.us-west-2.amazonaws.com",
      "headers": {
          "Host": "kinesis.us-west-2.amazonaws.com",
          "Content-Type": "application/json; charset=UTF-8",
          "X-Amz-Target": "Kinesis_20131202.PutRecord",
          "X-Amz-Date": "20180613T203123Z",
          "X-Amz-Security-Token": "&lt;session.token&gt;",
          "Authorization": "AWS4-HMAC-SHA256 Credential=&lt;access.key&gt;/20180613/us-west-2/kinesis/aws4_request, SignedHeaders=content-type;host;x-amz-target, Signature=ba20abb21763e5c8e913527c95a0c7efba590cf5ff1df3b770d4d9b945a10481"
      },
      "body": "\"{\\\"Data\\\":{\\\"task\\\":\\\"Get something working in kinesis\\\"},\\\"PartitionKey\\\":\\\"1\\\",\\\"StreamName\\\":\\\"MyKinesisStream\\\"}\"",
      "test": {
          "canonical": "POST\n/\n\ncontent-type:application/json; charset=UTF-8\nhost:kinesis.us-west-2.amazonaws.com\nx-amz-target:Kinesis_20131202.PutRecord\n\ncontent-type;host;x-amz-target\n508d2454044bffc25250f554c7b4c8f2e0c87c2d194676c8787867662633652a",
          "sts": "AWS4-HMAC-SHA256\n20180613T203123Z\n20180613/us-west-2/kinesis/aws4_request\n46a252f4eef52991c4a0903ab63bca86ec1aba09d4275dd8f5eb6fcc8d761211",
          "auth": "AWS4-HMAC-SHA256 Credential=&lt;access.key&gt;/20180613/us-west-2/kinesis/aws4_request, SignedHeaders=content-type;host;x-amz-target, Signature=ba20abb21763e5c8e913527c95a0c7efba590cf5ff1df3b770d4d9b945a10481"
      }
  </code></pre>
  
  <p>(the test key is used by the library that generates the signature, so ignore that)
  (Also there are probably extra slashes in the body because I pretty printed the response object using JSON.stringify).</p>
  
  <p>My question: Is there something I'm missing? Does Kinesis require headers a, b, and c and I'm only generating two of them? Or is this internal error an actual failure. I'm lost because the response suggests nothing I can do on my end. </p>
  
  <p>I appreciate any help!</p>
  
  <p>Edit: As a secondary question, am I using the <code>X-Amz-Target</code> header correctly? This <strong>is</strong> how you reference calling a service function so long as you're hitting that service endpoint, no? </p>
  
  <p><strong>Update</strong>: Followinh Michael's comments, I've gotten somewhere, but I still haven't solved the problem. Here's what I did: </p>
  
  <p>I made sure that in my <code>payload</code> I'm only running <code>JSON.stringify</code> on the <code>Data</code> property. </p>
  
  <p>I also modified the <code>Content-Type</code> header to be <code>"Content-Type" : "application/x-amz-json-1.1"</code> and as such, I'm getting slightly more useful error messages back.</p>
  
  <p>Now, my payload is still mostly the same:</p>
  
  <pre><code>var payload = {
      Data: JSON.stringify({ task: "Get something working in kinesis" }),
      PartitionKey: "1",
      StreamName: "MyKinesisStream"
  }
  </code></pre>
  
  <p>and my signer function body looks like this:</p>
  
  <pre><code> function signer({ url, method, data }) {
      // Wrapping with URL for piecemeal picking of parsed pieces
      const parsed = new URL(url);
  
      const [ service, region ] = parsed.host.split(".");
  
      const signed = sign({
          method,
          service,
          region,
          url,
  
          // Hardcoded
          headers : {
              Host           : parsed.host,
              "Content-Type" : "application/json; charset=UTF-8",
  
              "X-Amz-Target" : "Kinesis_20131202.PutRecord"
          },
  
          body : data,
      }, credentials);
  
      return signed;
  }
  </code></pre>
  
  <p>So I'm passing in an object that is partially serialized (at least Data is) and when I send this to the service, I get a response of:</p>
  
  <p><code>{"__type":"SerializationException"}</code></p>
  
  <p>which is at least <em>marginally helpful</em> because it tells me that my input is technically incorrect. However, I've done a few things in an attempt to correct this:</p>
  
  <ul>
  <li>I've run <code>JSON.stringify</code> on the entire payload</li>
  <li>I've changed my <code>Data</code> key to just be a string value to see if it would go through</li>
  <li>I've tried running <code>JSON.stringify</code> on Data and then running <code>btoa</code> because I read on another post that that worked for someone. </li>
  </ul>
  
  <p>But I'm still getting the same error. I feel like I'm <em>so close</em>. Can you spot anything I might be missing or something I haven't tried? I've gotten sporadic unknownoperationexceptions but I think right now this Serialization has me stumped.  </p>
  
  <p><strong>Edit 2:</strong></p>
  
  <p>As it turns out, Kinesis will only accept a <code>base64</code> encoded string. This is probably a nicety that the <code>aws-sdk</code> provides, but essentially all it took was <code>Data: btoa(JSON.stringify({ task: "data"}))</code> in the payload to get it working</p>