Hacker News: eastdakota

New comment by eastdakota in "Cloudflare outage on November 18, 2025 post mortem"

eastdakota — Wed, 19 Nov 2025 12:58:50 +0000

That’s not accurate. As with any incident response there were a number of theories of the cause we were working in parallel. The feature file failure was one identified as potential in the first 30 minutes. However, the theory that seemed the most plausible based on what we were seeing (intermittent, initially concentrated in the UK, spike in errors for certain API endpoints) as well as what else we’d been dealing with (a bot net that had escalated DDoS attacks from 3Tbps to 30Tbps against us and others like Microsoft over the last 3 months). We worked multiple theories in parallel. After an hour we ruled out the DDoS theory. We had other theories also running in parallel, but at that point the dominant theory was that the feature file was somehow corrupt. One thing that made us initially question the theory was nothing in our changelogs seemed like it would have caused the feature file to grow in size. It was only after the incident that we realized the database permissions change had caused it, but that was far from obvious. Even after we identified the problem with the feature file, we did not have an automated process to role the feature file back to a known-safe previous version. So we had to shut down the reissuance and manually insert a file into the queue. Figuring out how to do that took time and waking people up as there are lots of security safeguards in place to prevent an individual from easily doing that. We also needed to double check we wouldn’t make things worse. The propagation then takes some time especially because there are tiers of caching of the file that we had to clear. Finally we chose to restart the FL2 processes on all the machines that make up our fleet to ensure they all loaded the corrected file as quickly as possible. That’s a lot of processes on a lot of machines. So I think best description was it took us an hour for the team to coalesce on the feature file being the cause and then another two to get the fix rolled out.

New comment by eastdakota in "Cloudflare outage on November 18, 2025 post mortem"

eastdakota — Wed, 19 Nov 2025 12:36:20 +0000

There’s lots of things we did while we were trying to track down and debug the root cause that didn’t make it into the post. Sorry the WARP takedown impacted you. As I said in a comment above, it was the result of us (wrongly) believing that this was an attack targeting WARP endpoints in our UK data centers. That turned out to be wrong but based on where errors initially spiked it was a reasonable hypothesis we wanted to rule out.

New comment by eastdakota in "Cloudflare outage on November 18, 2025 post mortem"

eastdakota — Wed, 19 Nov 2025 12:33:08 +0000

Team has a good sense, typically. In this case, the names of the columns in the Bot Management feature table seemed sensitive. The person who included that in the master document we were working from added a comment: “Should redact column names.” John and I usually catch anything the rest of the team may have missed. For me, pays to have gone to law school, but also pays to have studied Computer Science in college and be technical enough to still understand both the SQL and Rust code here.

New comment by eastdakota in "Cloudflare outage on November 18, 2025 post mortem"

eastdakota — Wed, 19 Nov 2025 12:27:37 +0000

* published less than 12 hours from when the incident began. Proud of the team for pulling together everything so quickly and clearly.

New comment by eastdakota in "Cloudflare outage on November 18, 2025 post mortem"

eastdakota — Wed, 19 Nov 2025 12:24:22 +0000

Next time open your dev console in your window and look at how much is going on in the background.

New comment by eastdakota in "Cloudflare outage on November 18, 2025 post mortem"

eastdakota — Wed, 19 Nov 2025 12:22:54 +0000

We incorrectly thought at the time it was attack traffic coming in via WARP into LHR. In reality it was just that the failures started showing up there first because of how the bad file propagated and where it was working hours in the world.

New comment by eastdakota in "Cloudflare outage on November 18, 2025 post mortem"

eastdakota — Wed, 19 Nov 2025 01:43:06 +0000

Chicken burrito from Coyo Taco in Lisbon. I am not proud of this. It’s worse than ordering from Chipotle. But there are no Chipotle’s in Lisbon… yet.

New comment by eastdakota in "Cloudflare outage on November 18, 2025 post mortem"

eastdakota — Wed, 19 Nov 2025 00:38:43 +0000

Well… we have a culture of transparency we take seriously. I spent 3 years in law school that many times over my career have seemed like wastes but days like today prove useful. I was in the triage video bridge call nearly the whole time. Spent some time after we got things under control talking to customers. Then went home. I’m currently in Lisbon at our EUHQ. I texted John Graham-Cumming, our former CTO and current Board member whose clarity of writing I’ve always admired. He came over. Brought his son (“to show that work isn’t always fun”). Our Chief Legal Officer (Doug) happened to be in town. He came over too. The team had put together a technical doc with all the details. A tick-tock of what had happened and when. I locked myself on a balcony and started writing the intro and conclusion in my trusty BBEdit text editor. John started working on the technical middle. Doug provided edits here and there on places we weren’t clear. At some point John ordered sushi but from a place with limited delivery selection options, and I’m allergic to shellfish, so I ordered a burrito. The team continued to flesh out what happened. As we’d write we’d discover questions: how could a database permission change impact query results? Why were we making a permission change in the first place? We asked in the Google Doc. Answers came back. A few hours ago we declared it done. I read it top-to-bottom out loud for Doug, John, and John’s son. None of us were happy — we were embarrassed by what had happened — but we declared it true and accurate. I sent a draft to Michelle, who’s in SF. The technical teams gave it a once over. Our social media team staged it to our blog. I texted John to see if he wanted to post it to HN. He didn’t reply after a few minutes so I did. That was the process.

New comment by eastdakota in "Cloudflare outage on November 18, 2025 post mortem"

eastdakota — Wed, 19 Nov 2025 00:26:52 +0000

Can attest: not a single LLM used. Couldn’t if I tried. Old school. And not entirely proud of that.

New comment by eastdakota in "Cloudflare outage on November 18, 2025 post mortem"

eastdakota — Wed, 19 Nov 2025 00:08:11 +0000

That’s correct.

New comment by eastdakota in "Cloudflare outage on November 18, 2025 post mortem"

eastdakota — Wed, 19 Nov 2025 00:07:27 +0000

Because we initially thought it was an attack. And then when we figured it out we didn’t have a way to insert a good file into the queue. And then we needed to reboot processes on (a lot) of machines worldwide to get them to flush their bad files.

New comment by eastdakota in "Cloudflare outage on November 18, 2025 post mortem"

eastdakota — Wed, 19 Nov 2025 00:05:20 +0000

We don’t know. Suspect it may just have been a big uptick in load and a failure of its underlying infrastructure to scale up.

Cloudflare outage on November 18, 2025 post mortem

eastdakota — Tue, 18 Nov 2025 23:31:22 +0000

Related: Cloudflare Global Network experiencing issues - https://news.ycombinator.com/item?id=45963780 - Nov 2025 (1580 comments)

Comments URL: https://news.ycombinator.com/item?id=45973709

Points: 1465

# Comments: 916

New comment by eastdakota in "Unpacking Cloudflare Workers CPU Performance Benchmarks"

eastdakota — Wed, 15 Oct 2025 05:15:41 +0000

That definitely is one not-wrong conclusion.

New comment by eastdakota in "Unpacking Cloudflare Workers CPU Performance Benchmarks"

eastdakota — Wed, 15 Oct 2025 05:14:58 +0000

Blog run by the engineering team. I wouldn’t even know how to veto a post if I wanted to. Not in our DNA.

New comment by eastdakota in "Cloudflare Email Service: private beta"

eastdakota — Thu, 25 Sep 2025 17:46:48 +0000

I’m not going anywhere anytime soon.

Cloudflare's 2025 Annual Founders' Letter

eastdakota — Sun, 21 Sep 2025 17:51:37 +0000

Article URL: https://blog.cloudflare.com/cloudflare-2025-annual-founders-letter/

Comments URL: https://news.ycombinator.com/item?id=45325037

Points: 16

# Comments: 2

New comment by eastdakota in "Ask HN: Why is Cloudflare sending my US traffic to London?"

eastdakota — Wed, 27 Aug 2025 23:22:12 +0000

Free customers are served from the nearest location where we have capacity. If we’re capacity constrained then free customers will be the first to be rerouted to another facility with capacity. That typically only happens for a very narrow window during any day. It has nothing to do with load to your particular site. It has to do with a region’s capacity and a group of customers (e.g., FREE PRO BIZ ENTERPRISE).

New comment by eastdakota in "A look at Cloudflare's AI-coded OAuth library"

eastdakota — Sun, 08 Jun 2025 18:45:57 +0000

I agree with Kenton’s aside.

New comment by eastdakota in "Chaos in the Cloudflare Lisbon Office"

eastdakota — Tue, 18 Mar 2025 05:21:53 +0000

We’re in Munich, pretty close, and Lisbon, which is lovely. Unlikely Austria any time soon.