VPC Endpoint increases DynamoDB latency by 30%
(Erratum) VPC Endpoint increases DynamoDB latency by 30%

				Our reader Tom wrote in to tell me, that the latency for read requests to DynamoDB increased significantly after enabling a VPC endpoint a few weeks ago. Someone else reported a similar problem in the AWS discussion forums as well. Therefore, I started to investigate to write this article. After many hours of benchmarking, I came to the conclusion that using a VPC endpoint to connect to DynamoDB increase latency by 30% compared to connections through a internet gateway or NAT gateway. Shortly, after publishing the article, Petar send me a message via Twitter to tell me that something must be wrong with my benchmark. Unfortunatly, that was correct. I made a mistake when measuring the read latency for DynamoDB from EC2.
Therefore, I have depublished the original blog post. Instead you will find a post mortem analysis in the following.

The problem starts with Node.js and the AWS SDK. I was using the following script to measure the read latency for DynamoDB.
const AWS = require('aws-sdk')const dynamodb = new AWS.DynamoDB({apiVersion: '2012-08-10'});AWS.config.maxRetries = 0;async function benchmark() {  for (let index = 0; index < 10000; index++) {    let start = new Date().getTime();    let result = await dynamodb.getItem({      Key: {       "id": {         S: "1"        }      },      TableName: "benchmark"    }).promise();    let end = new Date().getTime();    console.log(`${index},${end-start}`);    await new Promise(r => setTimeout(r, 50));  }}benchmark();

With that code, requests through a VPC endpoint took about 10 ms - about 30% - longer than when connecting through an Internet gateway.
The problem with that Node.js code? Each DynamoDB API request create a new TCP connection. Doing so adds latency, and it seems like establishing a new TCP connection is taking about 30% longer when using a VPC endpoint.
However, you should not use the Node.js default in production. Instead, you need to tell the AWS SDK to reuse exsisting connections. To do so, I modified my code as described in Reusing Connections with Keep-Alive in Node.js.
const AWS = require('aws-sdk')const https = require('https');const agent = new https.Agent({  keepAlive: true});const dynamodb = new AWS.DynamoDB({httpOptions: {agent}});AWS.config.maxRetries = 0;async function benchmark() {  for (let index = 0; index < 10000; index++) {    let start = process.hrtime.bigint();    let result = await dynamodb.getItem({      Key: {       "id": {         S: "1"        }      },      TableName: "benchmark"    }).promise();    let end = process.hrtime.bigint();    console.log(`${index},${end-start}`);    await new Promise(r => setTimeout(r, 50));  }}benchmark();

With that modification I could no longer measure any significant differences between connecting to DynamoDB through an Internet gateway, NAT gateway, or VPC endpoint.
I’m sorry about my mistake. Michael and I are working hard on publishing high-quality content.
			