K8s cluster in peered VPC can't reach RDS cluster I have two K8s clusters in the same VPC that I want to connect to an MySQL
Aurora RDS cluster. One of the clusters can reach the RDS cluster just fine.
The other, however, cannot. I'll call these eks-cluster-working and
eks-cluster-broken.
I have a security group allowing traffic to the cluster:




Type
Protocol
Port range
Source
Description




MYSQL/Aurora
TCP
3306
sg-1 (eks-cluster-sg-working)
This rule works


MYSQL/Aurora
TCP
3306
sg-2 (eks-cluster-sg-broken)
This one does not




Both eks-cluster-working and eks-cluster-broken have the same default
security groups that AWS creates when making an EKS cluster and they are both
on the same K8s version (1.18). The one exception is that eks-cluster-working
has an extra security group for the load balancer service since it's hosting a
web service and the other one (eks-cluster-broken) is not. Both clusters
have outbound traffic rules that can reach 0.0.0.0/0.
I have zero firewalls, firewall policies and network firewall rule groups set
up in the region the clusters are hosted in.
This is the Terraform configuration I have set up for peering from my VPCs
to the default VPC.
resource &quot;aws_vpc_peering_connection&quot; &quot;default_to_environment&quot; {
  count       = local.num_environment_vpcs
  peer_vpc_id = data.aws_vpc.environment[count.index].id
  auto_accept = true
  vpc_id      = var.vpc_default_id

  tags = {
    Name      = &quot;Peer from default to ${data.aws_vpc.environment[count.index].tags[&quot;Name&quot;]}&quot;
    tf_module = &quot;vpc&quot;
  }
}

resource &quot;aws_route_table&quot; &quot;r&quot; {
  vpc_id = var.vpc_default_id

  # route = local.route_table_routes

  route {
    cidr_block = &quot;0.0.0.0/0&quot;
    gateway_id = &quot;igw-&lt;number&gt;&quot;
  }

  dynamic route {
    for_each = [for r in local.route_table_routes : {
      cidr_block                = r.cidr_block
      vpc_peering_connection_id = r.vpc_peering_connection_id
    }]
    content {
      cidr_block                = route.value.cidr_block
      vpc_peering_connection_id = route.value.vpc_peering_connection_id
    }
  }

  tags = {
    Name = &quot;default to environment VPCs&quot;
  }
}

Expected result
I expect eks-cluster-broken to be able to connect to the RDS cluster just
like eks-cluster-working can.
Question
What can cause this? Any suggestions as to how to go about debugging this?
For my Peering Connection, I went to:
&quot;Actions&quot; -&gt; &quot;Edit DNS Settings&quot; -&gt; &quot;Allow accepter VPC (vpc-&lt;id&gt;) to resolve DNS of requester VPC (vpc-&lt;id-default&gt;) hosts to private IP&quot;
and checked it. (It was unchecked before). I had to wait a little while before this took effect.