So, it looks like they were aware about such kind of issues and tried hard to mitigate them.

I sometime ask this person to hide the camera and they generally understand my feeling.

At build time, it generates a JSON file describing the image metadata and the layers data location. At runtime, it consumes this JSON file to stream layer data and image configuration to a destination. This is implemented by adding a new transport to Skopeo. Thanks to this implementation, nix2container doesn't need to handle all various destrination since this is managed by Skopeo itself.

Recently, we introduced a mechanism to also produce such kind of JSON file for the base image (see https://github.com/nlewo/nix2container?tab=readme-ov-file#ni...).

I'm pretty sure the added (not usptreamed yet) transport could be useful in some other close contexts, such as Bazel or Guix.

I'm the nix2container author and i will be glad to discuss with you if you think this Skopeo transport could be useful for you!

(btw, your blog post is pretty well written!)

I think this is a pretty important requirement to build trust in a team.

nix2container [1] is actually able to do that: you can explicitly build layers containing a subset of the dependencies required by your image. An example is provided in this section: https://github.com/nlewo/nix2container?tab=readme-ov-file#is...

For instance, if your images use bash, you can explicitly create a layer containing the bash closure. This layer can then be used across all your images and is only rebuild and repushed if this bash closure is modified.

> > pull in dependencies often results in massive image sizes with a single /nix/store layer

This is the case for the basic nixpkgs.dockerTools.buildImage function but this is not true with nix2container, nor with nixpkgs.dockerTools.streamLayeredImage. Instead of writing the layers in the Nix store, these tools build a script to actually push the image by using existing store paths (which are Nix runtime dependencies of this script). Regarding the nix2container implementation, it builds a JSON file describing the Nix store paths for all layers and uses Skopeo to push the image (to a Docker deamon, a registry, podman, ...), by consuming this JSON file.

(disclaimer: i'm the nix2container author)

[1] https://github.com/nlewo/nix2container

> Particularly with GitOps and Flux, making changes was a breeze.

i'm writing comin [1] which is GitOps for NixOS machines: you Git push your changes and your machines fetch and deploy them automatically.

[1] https://github.com/nlewo/comin

You can use Nix with GitHub actions since there is a Nix GitHub action: https://github.com/marketplace/actions/install-nix. Every time the action is triggered, Nix rebuilds everything, but thanks to its caching (need to be configured), it only rebuilds targets that has changed.

> How do you automate running tests and deploying to dev on every push

Nix is a build tool and it's main purpose is not to deploy artifacts. There are however a lot of tools to deploy artifacts built by Nix: https://github.com/nix-community/awesome-nix?tab=readme-ov-f...

Note there are also several Nix CI that can do a better job than a raw GitHub actions, because they are designed for Nix (Hydra, Garnix, Hercules, ...).

[1] https://www.chatons.org/en

For instance, on my NixOS machine, the Nginx configuration is not in `/etc/nginx` but explicitly provided and can then be known with ps:

$ ps aux | grep nginx

nginx: master process /nix/store/9iiqv6c3q8zlsqfp75vd2r7f1grwbxh7-nginx-1.24.0/bin/nginx -c /nix/store/9ffljwl3nlw4rkzbyhhirycb9hjv89lr-nginx.conf

It depends on the size of your Pex file (i don't think you mentioned it and sorry if i missed the info). With a Docker/OCI image, it would be possible to create a layer with only the Python files of your application (without deps and interpreter). (I say "would be possible" because that's currently not easy to achieve with Nix for instance.)

I think the culprit is more the GitHub Actions cache than Docker since it seems to be hard to get a clean cache management. I'm not sure about caching Docker image layers, but caching the Nix store with GitHub Actions is pretty complicated (not even sure it's possible): this means we have to download all required Nix store paths on each run, but i consider this is because of a GitHub Action cache limitation.

So, did you consider using another CI, which offers better caching mechanisms?

With a CI able to preserve the Nix store (Hydra[1] or Hercules[2] for instance), I think nix2container (author here) could also fit almost all of your requirements ("composability", reproducibility, isolation) and maybe provide better performances because it is able to split your application into several layers [2][3].

Note i'm pretty sure a lot of Docker CI also allows to efficiently build Docker images.

[1] https://hercules-ci.com/

[2] https://grahamc.com/blog/nix-and-layered-docker-images

[3] https://github.com/nlewo/nix2container/blob/85670cab354f7df6...

[1] https://github.com/nlewo/nix2container

And regarding OCI images, i built nix2container (https://github.com/nlewo/nix2container) to speed up image build and push times.

Note there is also an opened issue to restrict Nix fixed output derivation which would make impossible the buildGoModule "hack".

[1] https://github.com/NixOS/nixpkgs/issues/84826

[2] https://www.tweag.io/blog/2020-06-18-software-heritage/

https://www.gandi.net/en/domain/email

Currently, it supports the dockerTools layering algorithm and is designed to work with Guix [1] as well;)

[1] https://github.com/nlewo/nix2container/blob/065e5b108650ee4c...