Moving MongoDB off the NAT gateway

At the end of April the NAT gateways were processing about 437 GB a day. That is $19.65 a day, or close to $600 a month, for one line on the bill.

A little under half of it was the MongoDB cluster.

The database is in Atlas. The pods are in private subnets. The connection string points at a public hostname. So every query goes out through the NAT gateway, and every reply comes back the same way. Nothing is broken. It just works, and it bills you on both legs.

Why NAT is expensive

A NAT gateway charges three ways:

• $0.045 an hour per gateway, always on
• $0.045 per GB processed, in both directions
• normal internet egress on top

The middle one is the problem. It is 4.5 times what the same byte costs to cross an availability zone, and it applies on the way in as well as the way out. For traffic that never needed to leave AWS, that is money for nothing.

Flow logs will not tell you who

This is the part that cost me the most time, so it is worth saying plainly.

Turn on VPC flow logs with the default format and look at a NAT flow. The srcaddr field is the NAT gateway's own private address. The gateway already rewrote the packet. You learn that something went out to the internet. You already knew that.

The fields you need are pkt-srcaddr and pkt-dstaddr. Those are the real addresses, before translation. They are not in the default format. You have to ask for them:

resource "aws_flow_log" "vpc" {
  vpc_id                   = aws_vpc.base.id
  log_destination          = aws_s3_bucket.flow_logs.arn
  log_destination_type     = "s3"
  traffic_type             = "ALL"
  max_aggregation_interval = 60

  destination_options {
    file_format                = "parquet"
    per_hour_partition         = true
    hive_compatible_partitions = true
  }

  # pkt-srcaddr and pkt-dstaddr are the real peer, before NAT rewrites the header.
  # Without them a NAT flow has no name.
  log_format = "$${version} $${interface-id} $${srcaddr} $${dstaddr} $${srcport} $${dstport} $${protocol} $${packets} $${bytes} $${start} $${end} $${action} $${log-status} $${vpc-id} $${subnet-id} $${tcp-flags} $${type} $${pkt-srcaddr} $${pkt-dstaddr} $${region} $${az-id} $${flow-direction} $${traffic-path}"
}

Use Parquet with hourly partitions, and put a 30-day lifecycle rule on the bucket. Flow logs set to ALL get big fast. One 46-minute window here was 3.5 million rows.

The biggest talker

Once pkt-srcaddr is there, the question is just a GROUP BY. Add up inbound bytes by real source over an hour:

SELECT pkt_srcaddr,
       SUM(bytes) / 1024.0 / 1024.0 AS mb
FROM   vpc_flow_logs
WHERE  flow_direction = 'ingress'
  AND  pkt_srcaddr NOT LIKE '10.%'
GROUP  BY pkt_srcaddr
ORDER  BY mb DESC
LIMIT  10;

The top row was a single public IP. It sent 699 MB in under five minutes. The ASN lookup came back:

That was the database. Every result set was arriving over the public internet at $0.045 a GB.

The fix

Atlas can publish its cluster behind AWS PrivateLink. You create an interface endpoint in your own subnets and talk to it privately.

PrivateLink charges two ways, and they behave differently:

• $0.01 an hour for each zone the endpoint sits in. Fixed. Three zones comes to about $21.90 a month, whether a single byte flows or not.
• $0.01 a GB processed. Metered, against $0.045 a GB on NAT. So every GB moved across saves $0.035.

The fixed part is the catch. At low volume you pay $21.90 a month to save very little, and the endpoint is a worse deal than NAT. So the question is how much traffic it takes before the metered saving covers the fixed fee. Divide one by the other:

$21.90 a month  /  $0.035 saved per GB  =  626 GB a month  =  about 21 GB a day

Under roughly 21 GB a day the endpoint costs more than it saves. Over it, every extra GB is money back. The cluster was doing ten times that before breakfast, so there was nothing to think about.

Setting it up

Three resources, and the order matters. Atlas makes the endpoint service. You point an interface endpoint at it. Then you give the endpoint ID back to Atlas so it accepts the connection.

# 1. Atlas side. Returns the service name you need below.
resource "mongodbatlas_privatelink_endpoint" "atlas" {
  project_id    = var.atlas_project_id
  provider_name = "AWS"
  region        = "US_EAST_1"
}

resource "aws_security_group" "atlas_endpoint" {
  name        = "atlas-privatelink"
  description = "Pods to the Atlas endpoint"
  vpc_id      = var.vpc_id

  # A replica set needs 27017. A sharded cluster needs 1024-65535. Check first,
  # because getting this wrong shows up as an intermittent timeout.
  ingress {
    from_port       = 27017
    to_port         = 27017
    protocol        = "tcp"
    security_groups = [var.node_security_group_id]
  }
}

# 2. AWS side. One interface per app subnet, so a pod can always reach a local one.
resource "aws_vpc_endpoint" "atlas" {
  vpc_id             = var.vpc_id
  service_name       = mongodbatlas_privatelink_endpoint.atlas.endpoint_service_name
  vpc_endpoint_type  = "Interface"
  subnet_ids         = var.app_subnet_ids
  security_group_ids = [aws_security_group.atlas_endpoint.id]

  # Atlas gives you its own hostname, so there is no AWS private DNS name to take.
  private_dns_enabled = false
}

# 3. Give the endpoint id back to Atlas.
resource "mongodbatlas_privatelink_endpoint_service" "atlas" {
  project_id          = mongodbatlas_privatelink_endpoint.atlas.project_id
  private_link_id     = mongodbatlas_privatelink_endpoint.atlas.private_link_id
  endpoint_service_id = aws_vpc_endpoint.atlas.id
  provider_name       = "AWS"
}

terraform apply will sit still for a few minutes on step 1 and again on step 3. That is normal.

If you would rather do it by hand first, it is the same three steps:

# 1. Atlas: create the endpoint service, then wait for AVAILABLE
$ curl -s --digest -u "$ATLAS_PUBLIC_KEY:$ATLAS_PRIVATE_KEY" \
    -X POST "https://cloud.mongodb.com/api/atlas/v2/groups/$PROJECT_ID/privateEndpoint/AWS/endpointService" \
    -H 'Content-Type: application/json' \
    -H 'Accept: application/vnd.atlas.2023-01-01+json' \
    -d '{"providerName":"AWS","region":"US_EAST_1"}' | jq -r '.id'

$ curl -s --digest -u "$ATLAS_PUBLIC_KEY:$ATLAS_PRIVATE_KEY" \
    -H 'Accept: application/vnd.atlas.2023-01-01+json' \
    "https://cloud.mongodb.com/api/atlas/v2/groups/$PROJECT_ID/privateEndpoint/AWS/endpointService/$SERVICE_ID" \
    | jq -r '.status, .endpointServiceName'
AVAILABLE
com.amazonaws.vpce.us-east-1.vpce-svc-0a1b2c3d4e5f67890

# 2. AWS: the interface endpoint
$ aws ec2 create-vpc-endpoint \
    --vpc-endpoint-type Interface \
    --vpc-id "$VPC_ID" \
    --service-name com.amazonaws.vpce.us-east-1.vpce-svc-0a1b2c3d4e5f67890 \
    --subnet-ids "$SUBNET_A" "$SUBNET_B" "$SUBNET_C" \
    --security-group-ids "$SG_ID" \
    --no-private-dns-enabled \
    --query 'VpcEndpoint.VpcEndpointId' --output text
vpce-0a1b2c3d4e5f67890

# 3. Atlas: hand the id back
$ curl -s --digest -u "$ATLAS_PUBLIC_KEY:$ATLAS_PRIVATE_KEY" \
    -X POST ".../privateEndpoint/AWS/endpointService/$SERVICE_ID/endpoint" \
    -H 'Content-Type: application/json' \
    -H 'Accept: application/vnd.atlas.2023-01-01+json' \
    -d '{"id":"vpce-0a1b2c3d4e5f67890"}'

The last step is the connection string. Atlas gives the private route its own hostname. It is the same name with -pl-0 added to the cluster part, so the change is small and easy to miss in review:

# before — resolves to public IPs, goes out through the NAT gateway
mongodb+srv://appuser:PASSWORD@abc-cluster.9x7k2.mongodb.net/appdb?retryWrites=true&w=majority

# after — resolves to the endpoint interfaces inside the VPC
mongodb+srv://appuser:PASSWORD@abc-cluster-pl-0.9x7k2.mongodb.net/appdb?retryWrites=true&w=majority

Swap it in the secret, restart the pods, and check a flow log: pkt_srcaddr on those flows should now be a 10.x address instead of a public one.

What it did

The endpoint went in on 18 May. The applications moved on the 19th.

That is 194 GB a day gone. Half the NAT processing, $8.74 a day, about $266 a month.

The bytes did not disappear

They moved to a different meter. First full month on the endpoint:

The endpoint carries 210 GB a day. NAT lost 194 GB a day. That is close enough, and the gap is a month of normal growth between the two windows.

So the saving is not $266 a month. The endpoint costs $85 of its own. The real number is about $180 a month. Both figures are true. Only one of them is the saving.

Then the bytes got smaller

$0.01 a GB is cheap. It is not free, and the endpoint was still carrying 210 GB a day.

The MongoDB wire protocol can compress. It is off by default. Turning it on is one option:

// zstd needs @mongodb-js/zstd installed next to the driver. zlib is the fallback
// if the server does not offer zstd.
await mongoose.connect(uri, {compressors: ['zstd', 'zlib']});

You can also put ?compressors=zstd on the connection string and touch no code. It works in both directions and none of the queries change.

Compression went on 16 August. The endpoint dropped from 208 GB a day to 72 GB a day. That is 65% less traffic for exactly the same queries. BSON repeats every field name in every document, so it compresses very well.

The cost is CPU on both ends, and it barely registered. Pod CPU went up by one or two millicores, and on most pods the difference was too small to separate from normal noise. Worth watching on a heavier workload, but it did not show up as a problem here.

Three stages

The hourly charges belong in this picture too. Three NAT gateways cost about $100 a month before a single byte moves. The endpoint's three interfaces cost about $22 a month on the same terms.

The NAT gateways carry everything else, so their hourly charge stays on the bill either way. What moved is the rest of it: $266 a month of processing became $44 of endpoint, a saving of about $222 a month. If the database had been the only reason to run a NAT gateway, the $100 would have gone with it.

Per unit of data: a GB of query results used to cost $0.045 to receive. Now it costs $0.0069.

What this does not fix

The endpoint has one interface in each availability zone. That is correct, because a pod can always reach a local one. But nothing makes it prefer the local one. Private DNS is offPrivate DNS only works when the endpoint has an AWS service name to override, like s3.us-east-1.amazonaws.com. Atlas is a third-party service and hands out its own -pl-0 hostname in its own DNS zone, so there is nothing for AWS to override. The setting has to stay off, and with it goes any chance of the resolver preferring the local interface., so the driver sees all three addresses and picks one without caring where it is.

With three zones, roughly two thirds of queries leave the pod's zone. That lands on DataTransfer-Regional-Bytes at $0.01 a GB, charged on the way out and again on the way in.

Compression helped here too. Fewer bytes on the wire means fewer bytes crossing a zone, whichever meter picks them up.

What I would tell myself

Turn on pkt-srcaddr before you need it. Without it a NAT flow has no name.

External sources

Rates were current when this was written. AWS changes them, so check before you budget against them.

• NAT gateway pricing — the hourly charge and the $0.045 a GB processed
• PrivateLink pricing — the per-AZ hourly charge and the $0.01 a GB processed
• EC2 on-demand pricing — data transfer between availability zones, billed each way
• VPC flow log records — every field, pkt-srcaddr and pkt-dstaddr included
• Atlas private endpoints — the Atlas side of the setup
• MongoDB connection string options — compressors, and what the driver negotiates
• mongodbatlas_privatelink_endpoint — Terraform, Atlas side
• aws_vpc_endpoint — Terraform, AWS side