Taking over 330 GB a day of uncompressed HTML off the bill

In the five days to 21 August the cluster sent 333.18 GB a day out to the internet. At $0.09 a GB that is $29.99 a day, near enough $912 a month. It was the largest single line on the bill.

None of it was compressed.

The sites are server-rendered HTML behind a CDN. The CDN compresses what it sends to a reader, so pages arrive small and nothing looks broken. What leaves AWS is a different leg, and every page went out on it full size. Ask the cluster itself and it says so:

# the origin hostname: DNS straight to the load balancer, CDN not in the path,
# so this is the cluster's own response rather than the edge's
$ curl -sI -H 'Accept-Encoding: gzip, br' https://origin.site-a.example.com/
HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Vary: Origin

No Content-Encoding. The client asked for gzip and got 86 KB of plain HTML.

Two hops, two meters

A response crosses two metered hops on its way out.

The first is inside the region: app pod to gateway pod. The gateway runs one pod per node and picks a backend pod without regard to where it is, and the load balancer spreads requests evenly over three zones, so about two thirds of that hop leaves the zone it started in. That lands on DataTransfer-Regional-Bytes at $0.01 a GB, charged leaving one zone and charged again entering the next. A GB that crosses a zone costs $0.02.

The second is out of AWS: gateway pod to the CDN, at $0.09 a GB. Nine times the rate on the inner hop, which is why it went first.

The gateway was serving everything raw

The gateway is APISIX. Its gzip plugin was not running.

A client asking for identityThe HTTP name for no encoding at all: send Accept-Encoding: identity and the response comes back uncompressed. still gets the plain body, so both columns can be measured at the origin today. The left one is what everybody used to get:

80.7% of the bytes, gone for a block of configuration. Server-rendered HTML repeats the same shapes thousands of times a page, and gzip is very good at that.

Two places, or it does nothing

gzip is not an add-on. It is one of 92 plugins in APISIX's own default list, and the Helm chart ships plugins: [] so that those defaults stand. This cluster had set the field, and what it set was three plugins long:

 apisix:
   plugins:
     - prometheus
     - traffic-split
     - serverless-pre-function
+    - gzip

And the route:

plugins:
  - name: gzip
    enable: true
    config:
      types:
        - text/html
        - text/plain
        - text/css
        - text/xml
        - application/json
        - application/javascript
        - application/x-javascript
        - application/xml
        - image/svg+xml
      min_length: 1024
      comp_level: 4
      vary: true

The settings that matter:

• types defaults to ["text/html"] alone, which misses every JSON response. Images and fonts are left out on purpose: they are already compressed, so a second pass spends CPU to add bytes.
• comp_level defaults to 1. This traffic is billed by the GB, so a few points of ratio are worth more than the CPU. Level 9 doubles the CPU for another one or two percent, which is not.
• vary emits Vary: Accept-Encoding, so no cache in front can hand a gzipped body to a client that never asked for one.

What the gateway change did

It went live during the 17:00 UTC hour on 22 August. The load balancer's byte counter reads 14.2 GB in the 16:00 hour, 9.6 in the 17:00 and 3.3 in the 18:00.

250.05 GB a day at $0.09 a GB is $22.50 a day. Over an average month of 30.4 days, $684.

That sum is no check on itself: a flat-rate meter's cost is its volume restated.

The check is the load balancer, which counts bytes and requests separately and knows nothing of the bill. Requests did not fall: 12.96 million a day before, 13.52 million after. Bytes per request went from 26.6 KB to 6.4 KB. Traffic did not fall. The responses got smaller.

The hop behind it

The gateway now sends 14 KB to the CDN. The app still hands the gateway 86 KB.

That hop is metered at a tenth of the price, which is why it went second, but it is not small. In the three days before the change the cross-AZ line billed 1,417.71 GB a day, about 709 GB a day of real movement inside the region. Not all of it is this hop. Redis replicationEach Redis replica pulls a full copy of every write from its master. With replicas in other zones that traffic lands on the same cross-AZ meter, and it was the larger share of the line. is the larger share, and the database traffic on that meter had already been compressed on the wire.

A gateway cannot fix this. It compresses what it sends. The hop into it is already over by then.

Compressing in the app can go backwards

A byte saved on the inner hop is worth $0.02, and only two thirds of the time. A byte added on the outer hop costs $0.09, every time. If the app's gzip came out fatter than the gateway's, the change would lose on the expensive meter to save on the cheap one.

The margin is wide. One page drops 72 KB off the inner hop, about 48 KB of it crossing a zone at $0.02 a GB. Against 14 KB going out at $0.09 a GB, the app's gzip would have to run about three quarters fatter than the gateway's before the trade stopped paying. It is not fatter: both sides are zlib at level 4 with the default window and memory settings.

The real question is whether the gateway compresses that output a second time. It does not: the plugin drives nginx's gzip filter, and nginx does not double compress what arrives compressed from the server behind it. The wire agrees:

# the origin again, so the CDN cannot be the thing doing the compressing
$ curl -s -H 'Accept-Encoding: gzip' https://origin.site-a.example.com/ -o page.gz
$ gzip -l page.gz
  compressed uncompressed  ratio uncompressed_name
       14252        85801  83.3% page

14,252 bytes on the wire, 85,801 after one pass, and that second figure is exactly what the same URL returns to a client asking for identity. One layer, not two: the gateway is handing over the app's bytes untouched.

Three lines in the shared server

The apps are SvelteKit behind adapter-node, wrapped in a shared polka server that takes Connect middleware, so compression drops in:

import compression from 'compression';

const app = polka();

if (process.env.RESPONSE_COMPRESSION === 'true') {
  app.use(compression({level: 4, threshold: 1024}));
}

Level 4 matches the gateway. The 1 KB threshold is the library's default: below it, a gzip header costs more than it saves.

RESPONSE_COMPRESSION makes it a per-site switch, set in the deployment rather than the image, so turning it off is a redeploy and not a rebuild:

env:
  - name: RESPONSE_COMPRESSION
    value: 'true'

One site went first, the other two ten minutes later.

What the app change did

It went live at 21:52 UTC on 2 September for the first site and 22:02 for the other two.

337.21 GB a day at $0.01 a GB is $3.37 a day. Over 30.4 days, $102. The meter charges at both ends, so that is about 169 GB a day that stopped crossing a zone.

Two other instruments cover the same days. The load balancer served 14.25 million requests a day before and 14.37 million after, carrying 89.7 GB a day against 88.8 GB: traffic did not fall, and what leaves AWS did not move — the invariant the design rested on.

The EC2 network counters on the five nodes, which know nothing about either, went from 1,008.7 GB a day arriving to 776.5 GB. That is 232.2 GB a day less, a 23.0% fall, against a 23.8% fall on the billed line. Two systems counting different things, moving together.

The line keeps drifting down after 7 September. That is traffic, not the change: outbound falls by the same 12% over the same days.

The CPU never showed up

Gzip is not free, and it now runs on every response.

Node CPU averaged 37.9% across the five nodes in the three days before and 36.0% in the three days after, with traffic flat. It went down. Writing 14 KB to a socket instead of 86 KB is less work in the kernel than gzip adds in user space, so compression here is cheaper than the I/O it removes.

What both changes came to

Same pages, same HTML, and no reader sees a difference: the CDN already compressed the last mile.

Turning compression on at the edge of a system fixes the one leg you can see. Every metered hop behind it is still moving the full page, and on Kubernetes there is always one more hop than the diagram in your head.

External sources

Rates were current when this was written. AWS changes them, so check before you budget against them.

• EC2 on-demand pricing — data transfer between availability zones at $0.01 a GB in each direction, and the 100 GB a month of free egress.
• APISIX gzip plugin — every attribute and its default, including types defaulting to text/html alone and comp_level to 1.
• compression middleware — the level and threshold options, and what they default to.
• APISIX default plugins and the chart's plugins: [] — what a stock install enables, and what setting that field replaces.
• NGINX compression guide — that nginx does not double compress what a proxied server already compressed, which is why the gateway passes the app's bytes through.
• polka — the server the built SvelteKit output is wrapped in, and why Connect middleware works in it.