The idea is I could specify something like “I have this app that needs to pull data from a GPIO labeled as a ds18b20 and outdoors” and it takes care of scheduling my app, starting a ds18b20 reader, and networks them together. I never need to know if they’re on the same esp32, separate ones, or in a crazy alternate world the compute bit goes on a VM on a laptop with no GPIOs and the sensors go on a real esp32.

Having GPIOs be an addressable resource would be so cool. Like if nothing else, I could start a new esp32 project without wiring up a breadboard. Most of my stuff is some combination of the same 10 chips anyways.

- trustless code execution on someone's hardware (kube32 supports gpio permissions, amongst other stuff) - your standard Kubernetes cluster can now reach the physical world natively - OTA updates at arbitrary scale - fault-tolerance in the physical world, orchestration of workloads based on physical proximity or other sensor data

those are just a few that come to mind

If I count my test suite for the ISO, I have ~2500 unit tests (placement groups tests, Raft behaviour tests, disk perf tests, SigV4 crypto tests, DHCP tests, kernel rollback tests), ~110 fault-injection tests (latency, network, RAM), ~60 block-store failure tests (e.g. network interrupt while two disks try to sync), "chaos tests" (generating random event scenarios for an hour), bit flip tests, "fsync lies to me" tests, power cut tests, round-trip tests, "power cut while in the middle of sync between datacenters", SMART degradation tests, load tests, memory pressure tests, file descriptor resource starvation, NTP drift tests, metadata DB loss tests, DNS failover, ....

Then for security everything is turned off, kernel modules are off, eBPF is off, io_uring is off (lots of CVEs), user namespaces are off, most network and file drivers are off, /bin/sh is off, S3 endpoint is minimal, etc.

Most rules from aerospace are said to be "written with blood", some crash caused this rule to be legalized. But with ~40k lines you can go through each line and say "what if this fails". With a kubernetes cluster it's more or less based on hope and happy-path testing.

I also scraped other competitors for user complaints, esp. about data loss / sync engine bugs and in total I now have about 4000 tests for 40k lines of backend code and another ~1000 tests for the 15k lines for the sync engine code (about 1 test for every 15 lines).

I'm still not entirely confident (still need lots of beta testing), but a chance of data loss of .01% would be pretty high. It really depends on the amount of testing whether you need an on-call person at all and most orgs are more engineered towards "just hire someone to be on-call" rather than doing proper testing.

The take where in a couple years there will be a massive revert or that anyone cares about terraform indentation or a shitty yaml template is wildly optimistic.

I do hope you're right though, but I'm not seeing it from my side of the fence.

I think you’d be surprised if you look at the younger (post-2000-ish) companies. Specifically about this part:

> If you run simple workloads, it’s probably not necessary but then you’ll probably be paid for that.

You would be surprised how many workloads either are simple, or can be ham-fistedly shoved into a “simple” architecture.

Eg I’ve seen a lot of Postgres-on-k8s solutions that are basically just Postgres on bare metal with a bunch of useless abstractions. It writes to local disk, moving the pod to a new host is basically restoring from backup. Wrap it in some tool that ingests YAML and call it a day.

k8s is sort of the king of this. I hear people say they understand k8s and I can’t help but furrow my eyebrows. I don’t even pretend to, there are so many insanely complicated edge cases.

Eg if I’m sending UDP packets to a service and scale the deployment from 2 pods to 4 and the host for one of the new pods crashes and comes back up with a bad iptables rule, what are the range of potential behaviors k8s would show? What’s the difference between blocking say all traffic vs only traffic to one or two of the etcd instances? What if it’s blocking half the hosts in the cluster?

Storage has its own weird vagaries around single vs multi mounts and race conditions therein.

k8s is the king of “I don’t have to know that, the platform will do it”. I’m not sure I’ve ever heard anyone outside networking talk about whether a pod will saturate a NIC. Just chuck it in the cloud and let it figure itself out.

Machines (esp. Virtual Machines) are ultimately predictable and crashes don't "just randomly happen". But it's what this entire "enterprise cloud" culture has brought in, the idea that it's impossible to simulate all possible failure conditions of a cluster locally and therefore we shouldn't even try doing it. Maybe that's true at Google scale, but most companies are simply not, not, not Google-scale.

As a counter-example, I recently built a simple "cloud storage" system with a minimal Linux ISO: everything is stripped down to the absolute minimum to just boot, start a minimal S3 server and pull (encrypted and signed) update commands from an SQL "control plane" database. The entire userspace is ~50k lines of Rust (minus 10k init system) and I can simulate the entire VM locally and play with the "cluster" of VMs locally. The ISO is ~20 megabytes large and I can simulate entire datacenters "burning down", virtually, locally, before it happens for real. I can simulate everything: network failures, disk failures, slowdowns, latency, try getting Fable to hack it, etc.

But in the kubernetes "enterprise" world, I can't do any of this. First I need 10.000 values.yaml files, 10 add-ons for logging and metrics (because pushing metrics to some collection endpoint over HTTP is clearly PhD level complexity) and on top you need a service mesh to restrict random containers talking to each other because nobody knows anymore what services can talk to which other services. The results are constant outages because of misconfigurations and someone having to be on-call at all time.

Needing on-call at all (except for maybe security alerts, but those are usually not part of DevOps) is the result of this Google-influenced "crashes just happen" nonsense where "manager" culture is preferred over results (aka uptime).

If you run simple workloads, it’s probably not necessary but then you’ll probably be paid for that. Your company can also choose to outsource complexity to the cloud, and you’ll just be writing YAML. Then there’s also companies where you host everything yourself AND you have large/high workloads, then you still need people with understanding.

With your type of comments, it feels like you’re pretending a developer who writes WordPress websites does the same work as someone doing Erlang for instance. You’ll have different skill sets for every type of engineer, and they’ll all be necessary.

I don't think that's a bad thing, it makes infrastructure more consistent and less weird (assuming you understand k8s) but it is a very high level of abstraction and for a bunch of people is just a black box.

I have also come to the conclusion that, right now, AIs lack imagination. So designing human-AI collaboration that lets human imagination in the loop would be interesting.

If we’re going by water, land, etc as an argument against wasteful activities … Coming from the permaculture world, I can tell you just how wasteful the pre-AI world is. The ideas in the mainstream environmental-friendly practices are themselves very wasteful.

but so does React, Vue and Kubernetes and coding agents are crushing it.

I think if you started your company today, you would have a very different take.

> Because they have the resources to keep it up to date. And even they get it wrong sometimes

are these resources that require intelligence or something else?

If it could help you, we're using JIB. We're on Scala and so we're using sbt-jib. Zero issues, works like a charm, deployment (on k8s) is just not a problem for us. But I would expect that it should work well for Java too.

"No Topology-Aware Scheduling (TAS) support."

There is another blog talking about this coming soon.