We make Architect: a Kubernetes runtime that hibernates idle pods in place and wakes them in 50ms with TCP connections intact. Five engineers. You'd be the 6th.

If tracing an x86 instruction in the morning and hunting a control-plane race in the afternoon both sound fun, and you insist on measuring rather than guessing, this is the job.

Customers run Architect for workloads where cold starts hurt: real-time voice & video AI agents, long-warming JVM apps, stateful data services that can't be rescheduled cheaply. 1.0 shipped Q1 2026; you'd join mid-way through 2.0. Seed-stage, VC-funded, a few years of runway. Fully distributed across the Americas and Europe.

What you'd work on:

- Hibernation surface: containerd shims and CRUISE, our Zig-native CRIU replacement.

- Control plane: per-node DaemonSet streaming checkpoints; admission controller resizing hibernated pods in place.

- Networking and migration: eBPF/XDP at line rate; cross-node live migration to production; cross-cloud next.

You're a senior generalist. Years across the stack: assembly to frontends, hardware-near, comfortable in x86. Tests ship with the code, decisions get worked out in writing, and you measure rather than guess. Strong in Go; willing to use Zig, Rust, or C.

Bonus: eBPF/XDP, CRIU, Linux kernel internals, containerd, gVisor, live migration, or public writing in kernel/container/eBPF land. Strong systems depth and the willingness to pick up the rest is enough on its own.

Apply: https://loopholelabs.io/careers - we respond within the week (typically a few days).

This is what the GHA workflow currently looks like: https://github.com/thechangelog/changelog.com/blob/c7b8a57b2...

FWIW, you can see how everything fits together in this architecture diagram: https://github.com/thechangelog/changelog.com/blob/master/IN...

For everyone else reading this, we have been running https://changelog.com on Fly.io since April 2022. This is what our architecture currently looks like: https://github.com/thechangelog/changelog.com/blob/master/IN...

After 15 months & more than 100 million requests served by our Phoenix + PostgreSQL app running on Fly.io, I would be hard pressed to find a reason to complain. - Some deploys failed, and re-running the pipeline fixed it. - Early July 2023, 9k requests from Frankfurt returned 503s. Issue lasted 10 seconds. - While experimenting with machines, after many creations & deletions, one volume could not be deleted. Next day, the volume was gone.

That's about it after 15 months of running production workloads on Fly.io.

We mention about our Fly.io experience often in our Kaizen pod episodes, which we publish every ~2 months: https://changelog.com/topic/kaizen. For anyone curious, this is the episode in which we announced the migration: https://changelog.com/shipit/50. There is a detailed PR which goes with it: https://github.com/thechangelog/changelog.com/pull/407. We've been talking about our migration plan from apps v1 (Nomad) to apps v2 (flyd) recently: https://changelog.com/friends/2#transcript-138

I'm sorry to hear that many of you didn't have the best experience. I know that things will continue improving at Fly.io. My hope is that one day, all these hard times will make for great stories. This gives me hope: https://community.fly.io/t/reliability-its-not-great/11253

Keep improving.

The code in this repo tells the truth about what it is, and even shows how it works: https://github.com/thechangelog/changelog.com

I've just added YugabyteDB to my explore list.

A bunch of bare metal hosts run on Scaleway / Online, and different VMs & managed services run in Digital Ocean, Linode, AWS & GCP. I sometimes spin the odd bare metal instance on Equinix Metal (former Packet).

A diverse fleet means that there's always something new to learn and try out. A single large host would make me anxious, as no internet provider or power grid is 100% reliable and available. Also, software upgrades sometimes fail, and things get messed up all the time, which is when I find it most efficient to just start from scratch. A single host makes that less convenient.

Every approach has its pros and cons, which is why my main workstation is a 20 Xeon W with 64GB RAM & 1TB NVME : ). Yes, there is a backup workstation which doubles up as a mobile one meaning that it can work without power or hard internet for almost a day. Options are good ; )

crossplane.io comes closest afaik

> And is k8s the simplest possible abstraction? And if not - what is?

If you are asking about the simplest possible abstraction for container scheduling and orchestration, then I believe Nomad from HashiCorp or Docker Swarm are simpler. As for managed solutions with wide adoption in all types of environments and the largest investment to date, I am not aware of anything on par with K8S.

The devil is in the details, there is more to it than dynamic & static content, we are using Fastly, otherwise we couldn't serve all the traffic that we do.

The best part is that it's all public - https://github.com/thechangelog/changelog.com - and we welcome contributions, especially those that simplify our setup without compromising on resiliency and availability. I'm looking forward to yours ; )

For someone that is known as the King of Bash (self-proclaimed) - https://speakerdeck.com/gerhardlazu/how-to-write-good-bash-c... - and after a decade of Puppet, Chef, Ansible and oh wow that sweet bash https://github.com/gerhard/deliver - even if all my workstations and work servers (yup, all running k3s) are provisioned with Make (bash++), I still think that K8S is the better approach to running production infrastructure. The advantage to using simple and well-defined components (e.g. external-dns, ingress-nginx, prometheus-operator etc.) that adhere to a universal API, and are maintained by many smart people all around the world, is a better proposition than scripting in my opinion.

At the end of the day, I'm in it for the shared mindset, great conversations and a genuine desire to do better, which I have not seen before K8S & the wider CNCF. I will go on a limb here and assume that I love scripting just as much as you do, but go beyond this aspect and you will discover that it's more to it than "thin install scripts that deploy containers" (which are not just glorified jails or unikernels).

Concourse worked well for us, we didn't have any issues that were being enough to remember. You may be interested in this screenshot that captured the changelog.com pipeline from 2017: https://pipeline.gerhard.io/images/small-oss.png

I missed the simple Concourse pipeline view at first, but CircleCI improved by leaps and bounds in 2020, and the new Circle pipeline view equivalent is even better (compared to Concourse, clicking on jobs always works): https://app.circleci.com/pipelines/github/thechangelog/chang...

The Circle feature which I didn't expect to like as much as I do today, is the dashboard view (list of all pipeline/workflow runs). This is something that Concourse is still missing: https://app.circleci.com/pipelines/github/thechangelog

My favourite Circle 2020 feature is the Insights: https://app.circleci.com/insights/github/thechangelog/change.... Yup, we were one of the first ones to ask for it in 2019.

In 2021, I expect us to spend one migration credit on GitHub Actions, as a Circle replacement. Argo comes as a second close, but that requires an innovation credit which is more precious to us. Because we are already using GitHub Actions for some automation, it would make sense to consolidate, and also leverage the GitHub Container Registry, as a migration from Docker Hub. Watch https://github.com/thechangelog/changelog.com to see what happens : )

Other static files such as css, js, txt make sense to remain bundled with the app image, which is stateless and a prime candidate for horizontal scaling. Also, CDN caching makes small static files that change infrequently a non issue, regardless of their origin.

The managed Postgres service from Linode's 2021 roadmap is definitely something that we are looking forward to, but the simplest thing might be to provision Postgres with local volumes instead. We are already using a replicated Postgres via the Crunchy PostgreSQL Operator, so I'm looking forward to trying this approach out first.

CockroachDB is on my list of cool tech to experiment with, but that will use an innovation token, and we only have a few left for 2021, so I don't want to spend them all at once.

When failures happen, it's always a series of unfortunate incidents. When we've hit issues with Linode, we reached out and worked on what we can improve in our changelog.com setup, and discussed about the improvements that we can expect on the Linode end. Our common interest is a more resilient system, which requires a healthy collaboration, and Linode has been a great technology partner for us. Expect to see these write-ups on changelog.com as soon as these improvements have shipped, and we have hard data to support the claims ; )

I'm sorry to hear that things have not been as smooth for you on Linode. I hope that you will find an infra provider that you will be able to rely on and work with as we do. Not all collaborations will work out, and that's OK. It's also OK to be annoyed, fed up with the way things are and look for something different, something more suitable for you. My only ask is that you share your migration story with the changelog.com community. That is something that I would want to hear about.

Yes, we could have mitigated that entirely with CDN stale caching, but it was good to see what happens today, and then iterate towards better Fastly integration.

We have mostly sequential reads & writes (mp3 files) that peak at 50MB/s, then rely on CDN caching (Fastly makes us happy in this respect).

CDN caching is something that we are currently improving, which will make things quicker and more reliable.

The focus is on reality vs the ideal, and the path that we are taking to improving not just changelog.com, but also our sponsors' products. No managed K8S or IaaS is perfect, but we enjoy the Linode partnership & collaboration ;)

A good default cluster is 3 x r4.large with 100GB GP2 for RABBITMQ_MNESIA_BASE & pause_minority. For queues that need HA, a good default is ha-mode: exactly, ha-params: 2, ha-sync-mode: automatic. As for the Erlang version, we recommend 19.3.6.2 which has important fixes relevant for RabbitMQ. Today we recommend RabbitMQ 3.6.11, and 3.6.12 as soon as it ships.

In the past 6 months, I have been focusing on RabbitMQ stability and operability on AWS, GCP & vSphere. Can you tell me more about your RabbitMQ deployment lobster_johnson? This will help: https://s3-eu-west-1.amazonaws.com/rabbitmq-share/help-us-un...

I wouldn't mind moving this discussion to rabbitmq-users mailing list, so that it can benefit more in the RabbitMQ community.

Thanks, Gerhard

All things will break horribly if the conditions are right. It's unreasonable to assume that the things which work in [insert your current CI] will work in Concourse. It's still a new and relatively immature product, but it works well in most cases.

Half the secret to a good Concourse experience is not upgrading it in-place - stand up fresh deployments. The other half is gradually transitioning between Concourse deployments, because bad versions have been and will continue to be released - mistakes are only human. As long as you share the Concourse vision and are willing to keep up with the pace of change - not everyone can or wants to - then it's an amazing CI.

Concourse still makes me excited, even after many years of hard lessons, because it is a genuinely innovative approach to building better software. Most miss this, and I understand why, but give it time - the ideas behind it will mature and become the norm.

Even though Concourse can work really well, it's not always the best choice. Make it better if you can & want to, use something else if it's easier. There is no right or wrong, just preferences : )

More focus on the "why", less on the "how" http://thechangelog.com/ansible-docker/

I got frustrated enough with the Capistrano, Fabric, Puppi lot that I wrote a pure bash deployment tool with a fitting name htpps://github.com/gerhard/deliver.

Ansible on the other hand is something else though. There is some learning curve, agreed, but it's not as bad as awk or sed. And seriously, if you know your bash, you will know both awk and sed. I consider my shell scripting to be above average, and I've attempted a Docker orchestration pure bash tool https://github.com/cambridge-healthcare/dockerize, but Ansible just makes the same job easier. It's not everyone's cup of tea, but before you dismiss it, give it a real chance. I should know because I have initially dismissed it thinking that it's too complex, yet another Chef circus, yada yada, but trust me - it's worth it ; )