avro supports field renames though.

3. on second thought i believe you'd only have to deploy when you choose. the next build will force you to provide values (or opt into the default). so forcing inspection of construction sites seems good.

1. Dense json

Interesting idea. You can also just keep the compact binary if you just tag each payload with a schema id (see Avro). This also allows a generic reader to decode any binary format by reading the schema and then interpreting the binary payload, which is really useful. A secondary benefit is you never ever misinterpret a payload. I have seen bugs with protobufs misinterpreted since there is no connection handshake and interpretation is akin to 'cast'.

2. Compatibility checks

+100 there's not reason to allow breaking changes by default

3. Adding fields to a type: should you have to update all call sites?

I'm not so sure this is the right default. If I add a field to a core type used by 10 services, this requires rebuilding and deploying all of them.

4. enum looks great. what about backcompat when adding new enum fields? or sometimes when you need to 'upgrade' an atomic to an enum?

It goes into not only different isolation levels, but also some ambiguity in the traditional ACID definition.

I believe a 2nd edition is imminent.

What could be useful here is if postgres provided a way to determine the latest frozen uuid. This could be a few ms behind the last committed uuid but should guarantee that no new rows will land before the frozen uuid. Then we can use a single cursor track previously seen.

https://dagster.io/blog/fast-deploys-with-pex-and-docker

Computation is a physical process and any model we use to build or describe this process is imposed by us. Whether this model should include the concept of data (and their counterparts "functions") is really the question here. While I don't think the data/function concept is essential to modeling computation, I also have a hard time diverging too far from these ideas because that is all I have seen for decades. I believe Kay is challenging us to explore the space of other concepts that can model computation.

* It can be used as schema-less

* allows attaching metadata tags to values (which can serve as type hints[1]), and

* encodes blobs efficiently

I have not used it, but in the space of flexible formats it appears to have other interesting properties. For instance it can encode a symbol table making symbols really compact in the rest of the message. Symbol tables can be shared out of band.

[1] https://amazon-ion.github.io/ion-docs/docs/spec.html#annot

Comments URL: https://news.ycombinator.com/item?id=40948774

Points: 1

# Comments: 0

* https://en.wikipedia.org/wiki/A_Symbolic_Analysis_of_Relay_a...

Do you handle the case where the actual objects don't overlap but result of an aggregate query is still affected? For instance a `count(*) where ..` query is affected by an insert.

The other side of something like Spanner is the quorum-based latency is often optimized by adding another cache on top, which instantly defeats the original consistency guarantees. The consistency of (spanner+my_cache) is not the same as the consistency of spanner. So if we're back to app level consistency guarantees anyway, turns out the "managed" solution is only partial.

Ideally the managed db systems would have flexible consistency, allowing me to configure not just which object sets need consistency but also letting me configure caches with lag tolerance. This would let me choose trade-offs without having to implement consistent caching and other optimization tricks on top of globally consistent/serializable databases.

You may find other interesting articles linked from here:https://en.wikipedia.org/wiki/Jamie_Zawinski, eg https://www.jwz.org/gruntle/nomo.html

K8s may have some more controls for the "incremental deployment" case but I'm less confident about the isolation between pods to run user provided code.

There are two sides to the problem: the build and the deploy.

> it feels like just having a build machine with storage would immediately solve the problem, there would be no remote pulling. Is that not a solution?

Indeed, ephemeral builders are an issue and this would be an alternative solution. However doing this for an arbitrary numbers of users who want on-demand deploys for a variety of projects is non trivial. Most CI environments give you a "clean environment" at startup.

Even if the build takes 0 seconds, the other part is re-running this docker image (where 100s of MB is unchanged and say the final 1MB layer changed). You need to ship all these layers to a provisioned container and boot it up. Even this is possible to optimize by heavy caching and optimizing the container service - but that's an alternative solution with different trade-offs. With pex, we can update any docker image in-place so it is container service agnostic. It also works with our current service (Fargate) which has famously slow startup times.

Irrespective of the CI, I believe all cached Docker layers will need to be downloaded onto the build machine before it can be rebuilt.

Still, I believe it is possible to build and deploy faster even with a "docker image only" design and it's something we are still looking at. The question is what is the lower bound here - would be hard to beat "sync a file to a warm container and run it". Pex gives us a pretty good lower bound that is also container platform agnostic.

Firecracker is definitely very interesting. Would require more ops work for us to run bare metal EC2 (we currently use Fargate). IIUC reusing pre-existing environments would require us to share ext4 filesystems across the VMs. Not sure if antistrano helps here but will look into it.