Skip to content
Kerem Atam

Bug Chronicles #3: a 404 in Chrome and Firefox, but not in Safari

We wasted nearly 1.5 weeks as a team chasing a bug where the server returned 404 on a static file in Chrome and Firefox, but worked fine on Safari.

We couldn’t reproduce it with curl, which led to endless blame ping-pong between devs and infra guys.

Reproducing it

Repro steps were bizarre, and took so much time to find:

  • Toggle WiFi: it works
  • Toggle VPN: it works

Basically, touching the network interface would “reset” something in the browser request flow.

On some app versions, Chrome worked while Firefox still failed. No clue why.

What the browsers said

LLMs were mostly useless with the raw logs:

  • Chrome: (failed) net::ERR_BLOCKED_BY_ORB
  • Firefox: NS_ERROR_CORRUPTED_CONTENT

But once we managed to capture the weird repro patterns, Claude (with Opus 4.1) finally pointed us in the right direction.

The cause

It was HTTP/2 handling between browsers and infra. Disabling HTTP/2 on the browser always fixed it. But finding which proxy or server layer was responsible also took so much time.

Culprit, most probably: Traefik TCP Ingress. For some reason, it just couldn’t handle this edge case. We still don’t know the exact bug, but replacing Traefik solved it.

The weirdest part: Traefik was supposedly doing TCP passthrough (layer 4). But HTTP/2 runs at the application layer, multiplexing multiple requests on a single TCP connection. Somewhere in that handoff, things went sideways apparently.

Seeing it in the browser’s network log

If you need to sniff and analyze browser messaging in detail (similar to Wireshark) to better diagnose an issue, you can use net-export and netlog-viewer of Chrome.

Netlog view provides more detail than the dev tools tab. Using this, I’ve spotted the issue where an HTTP/2 request returned with an HTTP/1.1.

Capturing a log with net-export and opening it in netlog-viewer. 46 seconds, no sound.

I recorded the video with Screen Studio, a very easy to use tool for demos with a cool zoom-in feature.

Moral of the story

Our infra team introduced Traefik ~1.5 months ago, but we only discovered the bug much later.

If we had proper e2e tests with regression coverage, we could have caught this the moment it happened, blaming the right change immediately instead of wasting so much time.

And since we got it so late: if we had o11y monitoring more aggressively, we probably could have caught it sooner as well.

This post started as a thread on X and a follow-up post.



Comments

Comments are kept in GitHub Discussions. You need a GitHub account to write one.


Previous Post
Bug Chronicles #2: how I caused hundreds of double payments
Next Post
What an accessible PDF needs from the developer who generates it