the-mgi/notes

Tracing 200 GB a day to one Discord intent

7 min readawsnetworkingdiscordcost
TL;DR

The NAT gateways were processing 216.62 GB a day, about $296 a month. Almost all of it was coming in, not going out. I pulled the flow logs down from S3 and queried them with DuckDB. Five addresses were doing nearly all of it, and they all resolved to gateway.discord.gg. A shared module opened a Discord connection from every app pod, and one intent made Discord send every member status change to all of them. Taking that intent out took NAT to 14.75 GB a day.

From the start of June to the middle of August the NAT gateways processed about 216 GB a day. At $0.045 a GB, the rate in this region, that is $9.75 a day, near enough $296 a month. Check your own region before you reuse the number.

Nothing explained it. The database had already been moved off the NAT gateway onto PrivateLink. The sites sit behind a CDN, so user traffic does not come back through NAT. The pods talk to each other inside the VPC.

So something was pulling 200 GB a day in from the internet, and I could not say what.

It was one flag in a Discord client.

Getting the logs somewhere I could query them

With the default fields, NAT gateway flow logs obscure the original endpoint because srcaddr and dstaddr can represent the NAT gateway's network interface rather than the original Internet peer. The packet-level pkt-srcaddr and pkt-dstaddr fields preserve the original addresses.

The fields you want are pkt-srcaddr and pkt-dstaddr. They hold the real address at each end. You have to ask for them in a custom log format, which I wrote up in the flow log format that names the real peer. Same Terraform, same logs, a different answer at the end of it.

The logs land in S3 as Parquet files, split by hour. I copied them to my laptop and opened them with DuckDB:

shell
aws s3 sync s3://<flow-log-bucket>/AWSLogs/.../year=2026/month=05/ ./parquet/
duckdb

One query gives you the list of who is on the other end:

duckdb
WITH classified AS (
  SELECT
    bytes,
    CASE
      WHEN dstaddr IN ('10.42.0.10', '10.42.1.10', '10.42.2.10') THEN pkt_dstaddr
      WHEN srcaddr IN ('10.42.0.10', '10.42.1.10', '10.42.2.10') THEN pkt_srcaddr
    END AS external_ip
  FROM read_parquet('./parquet/**/*.parquet', hive_partitioning = 1)
)
SELECT external_ip,
       ROUND(SUM(bytes) / 1024.0 / 1024 / 1024, 2) AS gb
FROM   classified
WHERE  external_ip IS NOT NULL
  AND  external_ip NOT LIKE '10.%'
GROUP  BY external_ip
ORDER  BY gb DESC
LIMIT  10;

The three addresses in the CASE are the private addresses of the three zonal NAT gateway interfaces. Regional NAT gateways fill these fields in differently, so check yours before copying this. Each row is either a pod going out or the internet coming back. Which field holds the far end depends on which way the packet went, so you have to check both. Get it the wrong way round and every row is your own gateway again.

One thing about the byte counts. The gateway's interface sees every byte twice, once on the private side and once on the public side. AWS's own example shows four records for one round trip: pod to gateway, gateway to internet, internet to gateway, gateway back to pod.

The query keeps two of those four, and it takes both halves to do it. On the two private-side records the packet-level field holds the real internet peer, so the CASE picks it up. On the two public-side records that same field is the gateway's own address, so NOT LIKE '10.%' drops them. What survives is one record per direction, which is what AWS bills. A plain SUM(bytes) over the same interface, with neither the CASE nor the filter, comes out double.

AWS bills flow logs as vended logs, at $0.50 a GB delivered to S3.

Flow logs cost money

Logging every packet on a busy VPC runs to double figures of GB a day, and you pay for all of it. Work that out against your own traffic before you switch them on. Get your answer, then switch them off.

Wait longer than you want to

A forty six minute capture gives you a suspect. It does not give you a number.

You want a talker that stays big hour after hour. One spike tells you nothing, and if a spike is all you have, you will build a story around it. I let it run for days before I believed anything.

The five addresses

The top five rows were five addresses in the same block, each carrying about the same amount, every day:

top NAT destinations
162.159.130.234
162.159.133.234
162.159.134.234
162.159.135.234
162.159.136.234

Two checks name them. Both still work today. Forward DNS returns those same five:

shell
dig +short gateway.discord.gg A

And a TLS handshake to any of them returns Discord's certificate:

shell
echo | openssl s_client -connect 162.159.133.234:443 -servername gateway.discord.gg 2>/dev/null \
  | openssl x509 -noout -subject
# subject=CN=discord.gg

Over the last full week of May those five were 93% of everything the NAT gateways processed.

NAT processed per day — Discord gateway vs everything elseGB
Everything elseDiscord gateway
23 May23 May — Everything else 8 GB23 May — Discord gateway 151 GB15924 May24 May — Everything else 9 GB24 May — Discord gateway 168 GB17725 May25 May — Everything else 7 GB25 May — Discord gateway 169 GB17626 May26 May — Everything else 13 GB26 May — Discord gateway 179 GB19227 May27 May — Everything else 21 GB27 May — Discord gateway 162 GB18328 May28 May — Everything else 13 GB28 May — Discord gateway 154 GB16729 May29 May — Everything else 10 GB29 May — Discord gateway 162 GB172

Billed GB per day from flow logs, 23–29 May 2026. Discord averaged 163.6 GB a day of a 175.1 GB total.

Moving the database onto a PrivateLink endpoint had taken NAT down to 175 GB a day by the end of May. By June it was back over 200. Discord was the part that grew. It went from 163.6 GB a day in late May to 201.87 by the time it came out. Everything else stayed flat, at 11 to 15 GB a day.

One connection per pod

At this point I expected to find one process somewhere talking to Discord. That is not what was happening.

The backend is a monorepo. The Discord client is built in one shared module, and that module calls login(), which opens a connection to Discord and holds it open. The auth, user and subscription controllers all import that module. Those are normal request paths. The sites use them.

So every app pod that served a login opened its own connection. Each pod then got its own copy of everything Discord sent.

The stream did not arrive once. It arrived once per pod.

Watch out for this on Kubernetes

Anything that holds a connection open opens one per pod once it sits in shared code that normal request paths import. A Discord gateway, a Kafka consumer, a pub/sub subscriber, a websocket to a vendor. Your inbound traffic then scales with your replica count, not with your load, and an autoscaler quietly makes it worse. Check where the client is constructed before you count the connections.

GUILD_PRESENCES

The client was built with a list of gateway intents. An intent tells Discord which events to send you. Ask for fewer, get less.

One of them was GUILD_PRESENCES. That one makes Discord send a presence update every time a member of a server your app is in changes state. Online, idle, do not disturb, offline. It also fires when someone starts a game or plays a song, and when they change their name or avatar. On a big server this never stops.

Nothing in the application read presence data. Not one line.

The direction of the traffic gives it away:

DirectionGB/day
Inbound, from the internet197.51
Outbound, from the VPC19.11
Total processed216.62

Ten bytes in for every byte out. NAT charges both directions, so you do not have to ask for data to pay for it. It arrived, it cost $0.045 a GB, and it was thrown away.

Taking it off

One line came out of the intent list. It went out on 12 August.

shared/discord-client.js
 const client = new Client({
   intents: [
     Intents.FLAGS.GUILDS,
     Intents.FLAGS.GUILD_MEMBERS,
-    Intents.FLAGS.GUILD_PRESENCES,
     Intents.FLAGS.GUILD_MESSAGES
   ]
 });

 client.login(token);

NAT dropped the same day, and not gradually. The deploy landed around 04:00 UTC and the hourly figures fall off a cliff right there: 8.80 GB in the 03:00 hour, 1.21 GB in the 04:00 hour, 0.51 GB by 05:00. That is why 12 August reads as 45 GB rather than either level — four hours at the old rate, twenty at the new one. The table below starts the after window on the 13th for that reason. A month on, it has not drifted back.

NAT gateway — GB processed per dayGB/day
0100200300intent removed1 Jun12 Aug12 Sep1 Jun — 179 GB/day2 Jun — 195.6 GB/day3 Jun — 174.2 GB/day4 Jun — 186.8 GB/day5 Jun — 196.3 GB/day6 Jun — 218.8 GB/day7 Jun — 225.8 GB/day8 Jun — 225.7 GB/day9 Jun — 217.8 GB/day10 Jun — 234.3 GB/day11 Jun — 208.1 GB/day12 Jun — 229.9 GB/day13 Jun — 225.3 GB/day14 Jun — 190.6 GB/day15 Jun — 225.2 GB/day16 Jun — 213.8 GB/day17 Jun — 205.7 GB/day18 Jun — 214.2 GB/day19 Jun — 220.4 GB/day20 Jun — 218.7 GB/day21 Jun — 211.1 GB/day22 Jun — 212.6 GB/day23 Jun — 199.7 GB/day24 Jun — 201.2 GB/day25 Jun — 214.5 GB/day26 Jun — 205.5 GB/day27 Jun — 230.6 GB/day28 Jun — 196.8 GB/day29 Jun — 208.5 GB/day30 Jun — 205.6 GB/day1 Jul — 210 GB/day2 Jul — 243.4 GB/day3 Jul — 222 GB/day4 Jul — 234.2 GB/day5 Jul — 252.1 GB/day6 Jul — 236.4 GB/day7 Jul — 231.9 GB/day8 Jul — 225.1 GB/day9 Jul — 240.1 GB/day10 Jul — 248.7 GB/day11 Jul — 240.3 GB/day12 Jul — 252.5 GB/day13 Jul — 245.7 GB/day14 Jul — 223.7 GB/day15 Jul — 244.9 GB/day16 Jul — 237.9 GB/day17 Jul — 218.6 GB/day18 Jul — 224.5 GB/day19 Jul — 174 GB/day20 Jul — 217.6 GB/day21 Jul — 224.2 GB/day22 Jul — 220.1 GB/day23 Jul — 249.3 GB/day24 Jul — 221.7 GB/day25 Jul — 212.2 GB/day26 Jul — 229.5 GB/day27 Jul — 236.9 GB/day28 Jul — 212.2 GB/day29 Jul — 213.5 GB/day30 Jul — 208.5 GB/day31 Jul — 220.5 GB/day1 Aug — 189 GB/day2 Aug — 206.2 GB/day3 Aug — 210.8 GB/day4 Aug — 201.2 GB/day5 Aug — 208.8 GB/day6 Aug — 208.5 GB/day7 Aug — 222.1 GB/day8 Aug — 206.4 GB/day9 Aug — 198.4 GB/day10 Aug — 200.6 GB/day11 Aug — 179.4 GB/day12 Aug — 45.4 GB/day13 Aug — 12.2 GB/day14 Aug — 11.9 GB/day15 Aug — 24.9 GB/day16 Aug — 18.1 GB/day17 Aug — 16.1 GB/day18 Aug — 15.1 GB/day19 Aug — 15.1 GB/day20 Aug — 14.7 GB/day21 Aug — 14.9 GB/day22 Aug — 14.4 GB/day23 Aug — 13 GB/day24 Aug — 16.7 GB/day25 Aug — 14.9 GB/day26 Aug — 23 GB/day27 Aug — 17.3 GB/day28 Aug — 14.9 GB/day29 Aug — 25.1 GB/day30 Aug — 14.8 GB/day31 Aug — 20.6 GB/day1 Sep — 17.9 GB/day2 Sep — 14.7 GB/day3 Sep — 16.1 GB/day4 Sep — 14.5 GB/day5 Sep — 13.8 GB/day6 Sep — 9.40 GB/day7 Sep — 10 GB/day8 Sep — 8.20 GB/day9 Sep — 8.60 GB/day10 Sep — 8.80 GB/day11 Sep — 9.30 GB/day12 Sep — 8.30 GB/day

CloudWatch AWS/NATGateway, 1 June – 12 September 2026.

WindowNAT processedCost/dayCost/month
1 Jun – 11 Aug (before)216.62 GB/day$9.75$296
13 Aug – 12 Sep (after)14.75 GB/day$0.66$20
Difference201.87 GB/day$9.08$276

216.62 GB a day minus 14.75 leaves 201.87. At $0.045 a GB that is $9.08 a day. Over an average month of 30.4 days, $276.

That sum on its own proves nothing. NAT charges a flat rate per GB, so the cost is just the volume again in different units. Saying the cost fell because the volume fell is not a check.

The check is that two different instruments point the same way. In May the flow logs read actual packets and put 93% of NAT bytes on those five Discord addresses. In August the CloudWatch NAT metrics, which read no packets and know nothing about Discord, lost 201.87 GB a day of 216.62. The volume that left when the intent came out is the volume the packet capture had already pinned on Discord. Different months and different tools, so it is corroboration rather than the same measurement made twice.

External sources