http://docopt.org/

Make use of the usage string be the specification!

A criminally underused library.

If you haven’t tried it, it’s great!

I miss having lots of tools to reach for. Lots of different solutions, depending on where my company was and what they were trying to do.

I don’t think one T-shirt size fits all. But here are some specific things that annoy me.

Puppet had a richer change management language than docker. When I lost puppet, we had to revert back to shitty bash scripts, and nondeterminism from the cicd builds. The worst software in your org is always the build scripts. But now that is the whole host state! So SREs are held captive by nonsense in the cicd box. If you were using Jenkins 1.x, the job config wasn’t even checked in! With puppet I could use git to tell me what config changed, for tracked state anyway. Docker is nice in that the images are consistent, which is a huge pain point with bad puppet code. So it’s a mixed bag.

The clouds and network infrastructure have a lot of old assumptions about hosts/ips/ports. This comes up a lot in network security, and service discovery, and cache infrastructure. Dealing with this in the k8 world is so much harder, and the cost and performance so much worse. It’s really shocking to me how much people pay because they are using these software based networks.

The Hypervisors and native cloud solutions were much better at noisy neighbor protection, and a better abstraction for carving up workloads. When I worked at AWS I got to see the huge lengths the ebs and ec2 teams put into providing consistent performance. VMWare has also done a ton of work on QoS. The os kernels are just a lot less mature on this. Running in the cloud inside a single vm removed most of the value of this work.

In the early 2010s, lots of teams were provisioning ec2 instances and their bills were easy to see in the bill as dollars and cents. At my last company, we were describing workloads as replicas/gbs/cpus/clusters on a huge shared cluster. Thousands of hosts, a dozen data centers.

This added layer of obfuscation hides true cost of a workload. I watched a presentation from a large well known software service company say that their k8 migration increased their cloud spend because teams were no longer accountable to spend. At my company, I saw the same thing. Engineers were given the keys on provisioning but were not in the loop for cost cutting. That fell to the SREs, who were blamed for exploding costs. The engineers are really just not prepared to handle this kind of work. They have no understanding about the implications in terms of cost and performance. We didn’t train them on these things. But we took the keys away from the SRE’s and handed it to the engineers.

The debugging story is particularly weak. Once we shipped on docker and K8 we lost ssh access to production. 10 years into the docker experiment, we now have a generation of senior engineers who don’t know how to debug. I’ve spent dozens of hours on conference calls while the engineers fumbled around. Most of these issues could have been diagnosed with netstat/lsof/perl -pe/ping/traceroute. If the issue didn’t appear in New Relic, then they were totally helpless. The loss of the bash one-liner is really detrimental to engineers progress.

There is too much diversity in the docker base images and too many of them stuck. The tool encourages every engineer to pick a different one. To solve this my org promised to converge on alpine. But if you use a docker distribution, now you are shipping all of user mode to every process. I was on the hook for fixing a libc exploit for our fleet. I had everyone on a common base image, so fixing all 80 of my host classes took me about a few days. But my coworkers in other orgs who had hundreds of different docker images were working on it a year later. Answering the question, which LibC am I on became very difficult.

Terraform has a better provisioning/migration story. Use that to design your network, perform migrations. Use the cloud native networking constructs. Use them for security boundaries. Having workloads move seamlessly between these “anything can be on me hosts” make security, a real nightmare.

I left being an SRE behind when I saw management get convinced docker/k8 was a cancer treatment, a desert topping and a floor wax. it’s been five years and I think I made the right call.

The answer is not, “let an ai figure it out.”

That is legitimately scary.

Our industry is different than any other. The factory floor we work on is in our editors. The man on the assembly line building cars is a part of the machine. We are the people that builds the machine.

We don’t want to lock terms between management and labor, because as we’ve built up our tooling, we’ve changed the game repeatedly.

When I got started, I was writing c/c++/assembly. I had to write my own standard library for every project. I was allocating and freeing all my buffers. I had a qa guy, an ops guy, a dba supporting me.

Then I was a Java guy. We realized the DBA wasn’t needed anymore. I didn’t have to allocate and free. I could now use other people’s software through packages. I got way more productive. Made more money for the company, and I got paid more for it.

Then I was a python guy. We realized all this OO crap was a waste of time. We realized the QA guy could be replaced with better tooling and monitoring. The dev ops and cloud revolutions replaced my ops guy with APIs I could manage. I got way more productive. Made the company more money, and I got well paid for it.

Now we stand in front of the AI revolution. I don’t know what my job will look like. But it won’t look like what I’m doing now. I’m using copilot a lot, and I’m way more productive. I can turn around UI for the first time! I’m hoping these new Ai features we’ll make my company a lot of money. I should get some of it.

I’ve seen a lot of people age out when the technology changed. I’ve seen a lot of people make good money for their work. What scares me more than the next technology pushing me out, is an industry that stagnates around “the way we do it”. Today we are on the road to infinity. But if we stop moving forward we are on the road to stagnation.

The semantics of semantic versioning: you might be screwed

Blizzard please give us Warcraft 4.

Before they took all that money. Before they added all the crap people want but don’t need. It’s hard to do the fundamentals, especially when you take your eye off the ball.

http://www.softwarepreservation.org/projects/LISP/book/LISP%...

Once I get the state model fixed up, I was planning on writing some simple games. Perhaps snake or pong.

Comments URL: https://news.ycombinator.com/item?id=28353051

Points: 116

# Comments: 14

This article shows overhead for a disk heavy workload: https://vivait.co.uk/labs/docker-for-mac-performance-using-n...

He gets 7 second load time natively, but a 56 second load on docker with inconsistent drive link.

But on the Mac you have to pay the virtualization overhead to run Linux in a VM, then compound the docker overhead on top of that.

On MacBook Pro this is problematic because it eats battery and contributes to thermal problems.

The scripting environment managers have made big improvements in the past few years. Nvm/virtualenv/rvm give me 90% of the docker value prop with none of the performance impact or debug hassle.

Haskell/clojure/java all seem platform agnostic out of the box.

C/C++ is probably still a hassle with libs being platform specific.

At the start we had threads, pipes between threads, serial io, interrupts.

After we finished the project, I ported the system to standard UNIX system calls. Now I can tinker on it, and get full access to GDB/prints/profilers. I use signals, timers and pipes to simulate interrupts, clock and serial ports respectively.

Because I’m targeting UNIX it’s pretty easy to get started and build little toy experiments. I haven’t gotten it to compile on AVR for 10 years. But I have been maintaining it as a playground for new ideas.

Some recent additions are resumable functions (like python generators), A lisp user space, and a performance regression test suite.

https://github.com/andrewguy9/kernelpanic