title: Low Throughput of AWS Kinesis
body: <p>I use python's <a href="http://boto.readthedocs.org/en/latest/ref/kinesis.html" rel="nofollow"><code>boto.kinesis</code></a> module to write records to <a href="http://aws.amazon.com/kinesis/" rel="nofollow">AWS Kinesis</a>. The maximum throughput that is reached is about 40 puts/sec. However, according to the <a href="http://aws.amazon.com/kinesis/faqs/" rel="nofollow">Kinesis FAQ</a>:</p>
  
  <blockquote>
    <p>Each shard can support up to 1000 PUT records per second.</p>
  </blockquote>
  
  <p>So my current approach <strong>reaches only 4%</strong> what is theoretically possible, which seems terribly low. </p>
  
  <p>Does anyone have an idea how the throughput can be improved?</p>
  
  <p>Setup: The Kinesis Stream is an instance with one shard. The producer is on a dedicated AWS EC2 instance (t3.medium) in the same region as the Kinesis Stream. It creates strings of about 20 characters lengths and sends them to the Kinesis Stream via <code>boto.kinesis.Connection.put_record("my_stream", my_message)</code>.</p>
  
  <p>Simplified code:</p>
  
  <pre><code>from boto import kinesis
  import time
  
  connection = kinesis.connect_to_region(REGION)
  stream = connection.create_stream("my_stream", shard_count=1)
  time.sleep(60)  # wait a minute until stream is created
  
  for i in range(NUM_MESSAGES): 
      my_message = "This is message %d" % i
      connection.put_record(my_message, "my_stream", "partition_key")
  </code></pre>