Multi-repo appears to make teams faster (builds are faster! fewer merge conflicts!) but, like micro-services, they push complexity into the ether. Things like updating service contracts, library updates, etc. all become more complicated.

To your other point, Tilt is to development as ArgoCD is to deployment. Tilt enables on-demand, reproducible development environments that are sufficiently high-fidelity that you can often replace your shared and/or long-lived testing clusters.

With Tilt, I test my application using the same Kubernetes specs / Kustomizations / Helm Charts that you use to deploy into production. When it comes time to deploy my application, I supply these same specs / kustomizations / charts to ArgoCD.

Because I can reuse the specs for both testing and production, I enjoy far greater testability of my application, improving quality and time to market.

Want to create a Kubernetes secret? It's as simple as:

    load('ext://secret', 'secret_yaml_generic')
    k8s_yaml(secret_yaml_generic(...))

    load('ext://vault_client', 'vault_read_secret', 'vault_set_env_vars')
    vault_set_env_vars('https://localhost:8200','mytoken')
    my_foo = vault_read_secret('path/myfoo', 'value')
    my_bar = vault_read_secret('path/mybar', 'foobar')

I see that you also have docker-compose files -- are those for different tasks or for developer preference?

I'm also curious to understand why you have different build scripts for CI (`buildx`) vs local (regular docker build)? In our team, we use the same build processes for both.

In my case, I find that I prefer having higher fidelity and simpler service code by using Tilt to avoid mocks. It's also nice for frontend development because, using a Kubernetes ingress, you can avoid the need for things like frontend proxies, CORS and other development-only setup.

Tilt is a monitor process that builds and starts your services, with a hot-reload loop that rebuilds and restarts your services when the underlying code changes. The hot reload loop even works for statically compiled languages.

By default, Tilt is actually intended for local development using kind, minikube or other similar tooling. It supports developing against a multi-node cluster but it requires extra configuration and slows down iteration time.

- Service discovery

- Volume mounts

- Ingress and Certificates

- Metrics scraping and configuration

- Dashboards

It’s really quite powerful and replaces the need to mock things out with docker compose. If you’re deploying to Kubernetes, Tilt gives you the option to avoid “development-only” setups like docker compose.

I love how Tilt enables creating a local development environment that lets my services run the same in production , test and development. Greatly simplifies my service code and improved my quality.

In particular, I’d love to see Tilt be better around handling things like CRDs (there’s no way to mark a k8s_yaml as depending on a CRD being available, a frequent source of broken tilt up invocations).

Having said that, the first thing I do, when working on any new project, is to get “tilt up” working.

Things I’ve used for testing include: eBPF-based collectors for security and observability, data pipelines, helm chart development, and Kubernetes controllers. It’s very flexible and powerful for a wide range of development.

These are tools where the AI may tell you it’s doing one thing and then accidentally do another (I had an LLM tell me it would make a directory using mkdir but then called the shell command kdir (thankfully didn’t exist)). Sandboxing MCP servers is also important!

Do you have a limit in mind on how large you're willing to go? A brief bit of napkin math shows that, at 1KB per record, your largest view should fit within 100GB of memory. This is easily supported by Materialize and I can certainly understand if this exceeds your appetite!

If you're interested in reducing the size of the in-memory dataset, does your fact table have a temporal dimension to it? By default, Materialize stores data for all time (like a database) and you can write views in such a way that it will only materialize recent data (like a stream processor). Our co-founder Frank wrote a blog post[1] detailing how to do this.

I'm honestly not sure how TAIL-ing a non-materialized view would perform. Sounds like something that would be fun to test!

Happy to chat about this further in our community Slack channel if you have more questions.

[1] https://materialize.com/temporal-filters/

An example may not be necessary but it might also help clarify. Assuming you're using the psql client to run "TAIL WITH (PROGRESS)", the logical grouping for a single update will be a set of rows like the following:

  ...
  1608081358001 f -1 ['Lockal', '4590']
  1608081358001 f 1 ['Epidosis', '4595']
  1608081358001 f -1 ['Matlin', '5220']
  1608081358001 f 1 ['Matlin', '5221']
  1608081359001 t \N ['\\N', '\\N']
  ...

I find that this stream of rows is very easy to convert to a data structure "{timestamp, inserts[], deletes[]}" and this, in turn, maps naturally onto reactive APIs, such as React or D3. My blog post, linked above, delves into this in more detail. Hope this explanation helps!

I see that now -- makes sense!

> Wish we had some better database primitives to assemble rather than building everything on Postgres - its not ideal for a lot of things.

I'm curious to hear more about this! We agree that better primitives are required and that's why Materialize is written in Rust using using TimelyDataflow[1] and DifferentialDataflow[2] (both developed by Materialize co-founder Frank McSherry). The only relationship between Materialize and Postgres is that we are wire-compatible with Postgres and we don't share any code with Postgres nor do we have a dependence on it.

[1] https://github.com/TimelyDataflow/timely-dataflow [2] https://github.com/TimelyDataflow/differential-dataflow

Unfortunately, it turns out that recursively refreshing views still leads to surprising behavior. I think post summarizes the problem quite nicely: https://scattered-thoughts.net/writing/internal-consistency-.... If you cannot refresh all of the views, at a single point in time, then there will be internal inconsistencies in your dataset.

When looking at automatic refreshing, simple triggers and `LISTEN/NOTIFY` don't scale, as was mentioned in the comment regarding Hasura's multiplexing. I think, in the absence of incrementally maintained views, their multiplexing strategy is a good compromise for databases like postgres. However, it should be noted that continuous query / subscription of views is the exact scenario under which incremental computation will provide both lower latency and greater resource efficiency.