Thanks for the content-laden rebuttal!

The Dutch, ever innovative in trade, can lay claim as a bigger influence on the word “dollar” and the currency form itself, however, and colonial Americans traded regularly in Dutch daalders (we still pronounce it that way, unlike doh-LAHR/doh-LAHR-ehss for the Spanish varieties). Daalders themselves were descendants of Bohemian thalers, as were Spanish dollars. We just borrowed the neighboring dollars when the time came, probably due to our foreign policy environment at the time, trade with Florida, and so on.

There’s a bit of americentrism down the confusing line of thought, for what it’s worth.

When you said “the image” I thought you were looking at the right one, and I thought it odd you were off several orders of magnitude from what I assumed to be your misunderstanding. That explains that. I had to go back and find your figure.

The graph shows a spike to around $5,000 per day ($5 mil por día). The entire dashboard is in USD, presented in a Spanish locale. That is also why the dollar sign is suffixed, the months are not capitalized, and why May has a dot after it, because it is abbreviated there (mayo).

Every programmer should understand locales even if they do not speak the language.

To that end, I’m amazed it took that long for the FBI to take down the network in the article. The more people who are read in to criminal activity, the risk exponentially increases, as anybody who has been on either end of investigative leverage can tell you. I’m stunned one person in the early days of this scam, particularly when it started involving colorful people, didn’t flip as a bargaining tool for other things they were into.

As tomnipotent said, it’d make a cool movie.

As I said in my comment, those major customers (one personal experience, three anecdotally, eight or nine I’ve consulted with) have quietly ruled out Kubernetes, either by trying it or prying it apart and deciding not to try it. That feedback isn’t coming. At Borg scale, Kubernetes is very much considered a nonstarter.

Good. Because that was my point, but the verbiage “don’t disagree” says a lot. We agree, except for the timeline. We will all be dead before Borg is replaced with Kubernetes. You can take that to the bank.

Kinda weird to fire up a throwaway, presumably to conceal your Google credentials, then attack a Xoogler who used to work on Borg SRE (alongside sjh under Peter Dahl, and long enough my NDA has long since lapsed) and has run Kubernetes since it was able to OOM as I described, for spreading FUD. The term FUD implies that I don’t know what I’m talking about and I’m making shit up, while I’m one of the few people, including presumably you, who can actually coherently comment on both.

It can only span multiple clouds now because other clouds had to ship Kubernetes. Remember the timeline: hello world, we made a container thing. Now we offer it as a service. Now Amazon does too. Oh, we are now multicloud. Your rebuttals are quite disingenuous, and casting them with a mocking aspersion doesn’t sell your point. It makes you come across nothing like you intend.

Get back on your real username and stop being offended I criticized Kubernetes. I know I’m one of the few who does, but there are legitimate concerns, and sharing them toward a “why Borg isn’t going anywhere” point is a weird hill for you to die on.

I note you were careful to use the word descendant, instead of my successor.

What I mean is simple: Borg has borgmaster. Kubernetes approached the same concept like a Web application, and now Kubernetes has an entire SIG to play on the same field as Borg. It was a poor architectural decision, along with many others in Kubernetes, but I wasn’t discussing that. I was discussing why Google won’t replace Borg with it.

Without intimate knowledge of Borg, I can understand the successor discussion. With knowledge of what changed (i.e., was getting rid of borgmaster really that important to sacrifice that much perfwise?) I can’t even remotely fathom any purpose for Kubernetes other than what I’ve described. You, however, know far better than me. :)

There’s a meme out there, helpfully nudged along by Google, that Kubernetes is Borg “done right” and the successor. It’s even mentioned in this article. Neither of those things are true. Not even remotely. Please pass along to everyone to stop repeating the meme, because it distracts from Kubernetes’ true purpose, which is to lock people into GKE and force competitors to ship a Kubernetes runtime to compete. It’s partly the OpenStack playbook: if everybody runs the same platform, competition inherently drifts toward other aspects of the businesses, such as customer support. Seriously, I’m the only one who noticed the timing of Kubernetes and Google Cloud? Really? But Google shipped it, so now it has an ecosystem, and zealots who force entire teams onto it for zero upside and nonzero overhead, while the system that actually looks like Borg, the far superior to k8s HashiCorp Nomad, twiddles its thumbs with pretty much no mindshare.

For one, if Kubernetes were the successor to Borg, they wouldn’t have hobbled it architecturally as much as they have by marrying it to both Docker (kinda) and etcd, and deciding in the beginning to do every cluster mutation via external consensus in etcd, because you know, that’s a great idea and a classic Google design. Remember when Kubernetes pushed all job state changes through consensus and a flapping job could OOM etcd? I do too. Someone cynical could argue its fundamental architectural limitations are intentional. (I would argue simply that it didn’t have Paul Menage and most of the other names on the Borg paper working on it, to my knowledge.) I hear keyboards getting angry to yammer about how it just works. Not at seven-digit machine scale, it doesn’t, and never will. I’m happy it works for you. It’s a toy for a large fleet, which I’ll revisit in another point. Everyone I am aware of at Borg or Mesos scale has ruled out or failed with Kubernetes. No, really.

Relatedly, if Kubernetes were the successor to Borg, it’d be in C++. It just would be, and that’s not a language flame war. Ever wonder what percentage of systems at Google are C++? Ever wonder what that number does when you qualify “infrastructural”?

For two, Google containers aren’t an entire operating system, unlike Docker without crazy gymnastics. Seriously, this paragraph could be an essay. To paraphrase Jeremy Clarkson, Docker looks like a proper container system described over a blurry fax. Maybe we do need seventy probably-not-deduplicated copies of getty on every machine, and I’m yelling at clouds. I doubt it, and it reeks of “disk is cheap, fuck it.”

For three, Kubernetes is several orders of magnitude behind Borg and Omega, if that’s still alive, in terms of scheduling performance and maximum “cluster” size (I quote cluster because Google identifies a Borg unit as a cell, and a cluster means something else). This is not fixable in Kubernetes, in my reasonably informed opinion, without doing consensus differently. To my point, Borg does consensus and cluster state much differently, and you know what? It’s fine. Anybody who has used fauxmaster will back me up on that, and John Wilkes even said yup, every time we hit a limit we manage to double it with no end in sight. Why would that suddenly change? etcd remains Kubernetes’ Achilles heel, and this is why messaging around Kubernetes has gravitated toward smaller, targeted clusters. Bonus: if you find a bug in etcd make sure it loudly affects Kubernetes so it gets properly prioritized. Double bonus: someone was brave enough to suggest Consul in #1957 and children. Go read and sigh.

For four, when’s the last time you ran a 10,000+ node MapReduce on Kubernetes? Surprise, the underpinnings of Borg handle both batch and interactive with the exact same control plane, which is where the billions of containers a day number they occasionally talk about comes from. I mean, several JARs of glue might get you to Hadoop scheduling via Kubernetes, but that’s a much different animal than the platform thinking in terms of jobs with different interactivity requirements.

For five, half of Borg is the shit around it. Borg works because everything behaves the same. Everything is a Web server. Everything exposes /statusz. Everything builds and monitors the same way. Everything speaks the same RPC to each other. All of this is implemented by forcing production systems at Google to choose from four (as of my tenure) languages which are well tended and manicured by hundreds of people. Google has a larger C++ standard library team than many startups have engineers. Borg works because apps let it work. They’re not black boxes. Unlike Kubernetes.

Which brings me to point the sixth, which is that the reason you haven’t seen open source Borg is (a) they’re not moving off it, like, ever (I’d bet my season tickets on it), because significant parts of every production system and tool would have to change and (b) they can’t unravel Borg and the rest of the google3 codebase, because it’s so fundamental to the Google ecosystem and half of Borg’s magic is wrapped up in other projects within Google which they aren’t keen to show you.

Link to this answer next time anyone gets tempted to relay what they’ve heard about Borg and Kubernetes, please. For years I’ve watched this tale evolve until it’s barely recognizable as factual. Saying Kubernetes is Borg’s proper successor not only drastically insults Borg and the hundreds (thousands?) who have worked on it, it also calls to mind thinking of a cotton candy machine as the successor to the automobile. They’re that different.

Relatedly, in case you don’t know this, never think you can call a marine a sailor based on the lineage you’re discussing here. Soldier is also only an appropriate term for someone in the Army, and there are countless films that screw this up. It’s less about the service and more of an identity.

The gap between engineering and regular human beings continues to widen. Passwords remain difficult for people, but don’t let me stop the whiteboarding of a solution to the problem that will undoubtedly involve machine learning, Facebook OAuth, and GKE at some point. Add blockchain and pitch it to USDS before Thiel guts them and you’ve got yourself a hot party!

I’m boxed in a corner. If I explain it one way, you’ll interpret a Luddite sentiment. If I explain it another, you’ll interpret an affinity toward socialism. Yet another, and you’ll conclude I’m mentally challenged. I’m still working on vocalizing my detest for the valley and this audience in a form which is productive as opposed to punitive, so you’ll have to check back with me when I’ve matured that.

Like half a cringeworthy Cali startup’s engineers spending their morning at Sightglass discussing an article about a password manager software being bought and lamenting whatever that future holds for Windows support, you mean? While their white male founder is parading around saying he’s making the world a better place with his singular vision about a grand Kubernetes mumble mumble, but entertaining a slow progression of M&A offers to cash out and just dump the entire company on Red Hat or Google to sort out because “serial entrepreneur” is a better title than ever hitting one nail or dropping one load of fries? Meanwhile, deadlines keep missing because the engineers are busy commenting on this article as noted, and arguing about JavaScript frameworks, and “WFH” every Tuesday and Thursday because commuting like everyone else for $175,000 cash is just too hard for precious snowflake engineers (seriously, have you smelled BART?), and God help anyone who schedules a 10am meeting. Remember, machine learning is this cycle’s password to get a YC check, and start your own journey to enriching your wealth and throwing a bunch of starry-eyed people who mystifyingly trusted you into whatever horrible business unit of whatever horrible corporation writes you the check you’ve earned on account of your complete lack of professional experience, biological fortunes, and who you’ve done cocaine and/or Bitcoin with most recently. Oh, and you were so busy making money, you didn’t stop to notice you’ve literally handed every nation state in the world citizenry command and control capabilities they only could have dreamed of before this stupid industry found a whiteboard and daringly ventured, “what if we put millions of dollars into letting Justin Bieber type 140 characters to fans, because that could never go wrong?”

Did you mean that kind of meme? If not, I may have some bad news.

You added an unnecessary level of indirection. The startups don’t have memes. They are the meme.

These are lives you’re talking about here, kids, kids that happen to be currently sharing the same planet as you. Human beings are not figures on a balance sheet, and I lament anyone’s soul that would dispute me on that point. The pillars of this community sit on billions and get hospitals named after them, but some brown kids in a cave are a theoretical exercise to think about economics? They have memories. Families. Hopes. Should we debate your worth? What’s the figure where you are no longer considered worthwhile?

If I can’t meet you on something as fundamental as every life is worth saving, we simply aren’t having the same conversation, and I’ll be honest, your side gives me pause. And I see it far too often among people here who simply consider themselves pragmatic. Many throughout history have convinced themselves of their pragmatism, and we have all paid for it for centuries.

What unfortunate times to live in where the scions of high tech debate the value of individual lives in a far flung part of the world. This thread should disgust.

Aside from all of that, good analysis.

Aware iPhone owners literally have the “oh, they’re green, looks like I have yet another plaintext compromise in my life and they’ll never figure out Signal” conversation with themselves every time they exchange numbers with someone new, and I’ve met more than one person who has remarked negatively to the person’s face that they are using SMS, including in a dating scenario as a dealbreaker. You might be the only person who likes it, aside from misguided parts of the Android community that praise SMS roughly like you do when discussing the lack of something like iMessage in that ecosystem, all without realizing the surveillance calculus you’ve correctly established is the real reason the carriers tie Google’s hands (even beyond Google going through messaging systems like socks).