https://youtu.be/epORYuUKvZ0?si=gkVBgrX2VpBFQAk5

OP talks in the summary about the importance of understanding the process. It's very much true, but you need to understand not only the process your tooling provides, but also what's going on in the background and what hasn't been taken care for you by your tools. I'm afraid these things are rarely understood about hot upgrades, even by experienced Erlang engineers.

But via citogenesis, the coati really became also known as the Brazilian aardvark. So the original claim is true, and this wasn't really citogenesis after all. More like self fulfilling prophecy.

Deploy time: 2022-11-10 06:00

Deploy point: 5.31° S , 18.138° W

Simulation length: 9 days

Deploy time: 2022-11-10 03:00

Deploy point: 23.795° S , 133.047° E

Simulation length: 10 days

When it comes to node names in the schema, you have to change them, but there's actually an example just about this piece of task in the docs: https://www.erlang.org/doc/apps/mnesia/mnesia_chap7#backup,-...

It would have been much more elegant to add some crazy overpriced option to your pizzas where you could make a huge margin. Like "gift wrap +$100" which would mean putting a $0.01 ribbon around the box. You could make much more money this way from Doordash with less hassle than switching certain orders to plain dough.

About 10 years later Klarna started to change its system architecture (which makes total sense, since by that time it grew to a successful unicorn, with a way more complex business than what it had at launch time).

First, instead of shovelling every new feature into the monolith, it introduced new (mostly micro-)services around it. This model is successful, and most of these new services are written in Java, but there are also Erlang ones and other, even some "exotic" ones, like Haskell. (Also, through a series of acquisitions, Klarna got a bunch of tech from other companies, which, you can imagine, were written in whatever language those other companies used.)

There was also an attempt to replace Kred, the old Erlang system, with a new one (happened to be written in Java), and this project indeed was a failure in the sense that Kred is still around and nobody wants to decommission it nowadays. But the intended replacement system still proved useful, so it's not really a failure from that system's developers point of view.

However, when it comes to metrics in particular, I'd say Kafka was still a soft dependency. The reason we lost our metrics during the incident was a scheduler lock-up blocking the collection of VM-level metrics. It's just coincidence that the scheduler lock was caused by brod in this case. Otherwise metrics would just flow to Graphite directly, never interacting with Kafka.

When it comes to the complete monitoring solution around Kred, there was one bit of it depending on Kafka: System monitor (https://github.com/klarna-incubator/system_monitor). This tool is exporting data that doesn't fit well into Graphite or Splunk, so we store it in Postgres instead. Due to pure laziness it was at the time of the incident not writing directly to Postgres though, but was just pushing the data to Kafka. (The reason is that we already had a service available to push data from Kafka to Postgres.) After the incident we eliminated Kafka from the path. I didn't mention this work in the post because it was only marginally related.

Similarly, you wouldn't know which technology would work in this world and which not. Maybe alchemy would be real, and you could develop it to mass produce rare materials. Maybe it would turn out to be fake science only. Similarly, magic and religion may work as either basic mind tricks and psychology that enlightenment would mostly cancel out, or be part of the reality of the world, and you could get gods fighting on your side Greek mythology style, or wizards casting spells even deadlier than tanks and nukes.

I guess this system would be already a bit too hard to implement, but if I could keep wishing freely, it would be awesome if you could actually govern by writing whatever law you want. So you wouldn't just click a button to switch from feudalism to theocracy or communism, but you would actually have to come to an agreement with power figures (or classes) in your society on how your state would work. You could grant rights to tax trade routes in exchange of doing military service for example. And later you would need support from some other group if you would like to abolish this system.

> an ISA design that contains no unbounded local looping primitives

Actually, most of the Beam instructions are non-O(1). For example since integers are unbounded in Erlang even simple arithmetic (+, -, etc.) may turn out to be a non-constant time operation (even though most of the time your integers will fit into a single machine word, so it'd rarely be a problem). But there's also a built-in for appending lists (++), which obviously contains an unbounded loop in it.

The solution is that these instructions are written so that they do work on chunks of data (e.g. ++ may process 1000 elements of a list in a chunk) after which they increment the reduction counter and possibly schedule out the process.

> there's no other way to do loops given the ISA constraints

The Beam ISA allows looping. You can even write a hand-crafted loop that will never increment the reduction counter and thus will deadlock a scheduler. But the Erlang compiler will never generate such a loop for you.

On the ISA level there are only labels where you can jump to. You can jump forwards and backwards. So you could implement a language that offers loops and still compiles to Beam. However, since jumps don't increment the reduction counter, you would either risk your loops breaking the fair scheduling of processes, or you would have to ensure that the loop body contains an operation that increments the reduction counter and allows the scheduler to suspend the process.

> This atomicity of bytecode basic-blocks is what guarantees that actors can be hard-killed without corrupting the abstract-machine scheduler they run on

Well, it is of course important that you don't interrupt the scheduler at an arbitrary point, midway executing an opcode. But there are no atomically executed bytecode blocks. Actors are free to kill not because they would run their code in uninterrupted atomic blocks, but because they don't share state (their heap) with each other. So if you have an actor that holds e.g. a binary tree, and it is half way into inserting a value into the binary tree when you kill it, it may leave the binary tree in an inconsistent state, but that doesn't matter, because no one else have access to this data structure: it lives on this process' own heap.

When processes use shared resources (such as ETS tables, files or a gen_server process) and they are killed, they may very well leave that shared resource in an inconsistent state, just not on the VM layer, but on the application logic layer. So the file will still be usable as a file, but it may contain corrupted data for example.

> The JVM doesn't have this atomicity, and so you can't hard-kill a Java thread without corrupting the JVM. Instead, you can only softly "interrupt" threads.

If you would port Erlang to the JVM, that would be the least of your problems. The compiler could just insert code to check for these signals every now and then. I go further: if you'd run Erlang code (and only Erlang code) on the JVM, it wouldn't even matter that you don't have separate heaps. Every process would only use a separate part of the shared heap, so they couldn't tip on each other's toe. The GC could take care of the rest as usual.

I think there are two real issues with porting to the JVM:

* Mapping an Erlang process to an OS thread would only work up to some reasonably low number of Erlang processes. After that you'd have to switch to a green thread model with schedulers, which is a lot of work to implement. * The Beam put a lot of effort into making the VM scale well to a lot of schedulers. Things like how to implement a message box where 100+ schedulers can concurrently push messages to. You'd probably have to implement similar optimisations for the data structures you'd use for message boxes, ETS tables etc. on the JVM too.