It amazes me how many people will install github apps that have wide scopes, primarily write permissions to their repositories. Even with branch protection, often people will allow privilaged access to their cloud in github actions from pull requests. To properly configure this, you need to change the github oidc audience and that is not well documented.

When you enquire with the company who makes an app and ask them to provide a different app with less scope to disable some features which require write, they often have no interest what so ever and don't understand the security concerns and potential implications.

I think github need to address this in part by allowing more granular app access defined by the installer, but also more granular permissions in general.

I have found Datadog to be, by far hands down the best developer experience from the get go, the way it glues the mostly decent products together is unparalleled in comparison to other products (Grafana cloud/LGTM). I usually say if your at a small to medium scale business just makes sense, IF you understand the product and configure it correctly which is reasonably easy. The seamless integration between tracing, logging and metrics in the platform, which you can then easily combine with alerts is great. However, its easy to misconfigure it and spend a lot of money on seemingly nothing. If you do not implement tracing and structured logs (at the right volume and level) with trace/span ids etc all the way through services its hard to see the value, and seems expensive. It requires some good knowledge, and configuration of the product to make it pay off. The rest of the product features are generally good, for example their security suite is a good entry level to cloud security monitoring and SEIM too.

However, when you get to a certain scale, the cost of APM and Infrastructure hosts in Datadog can become become somewhat prohibitive. Also, Datadogs custom metrics pricing is somewhat expensive and its query language cababilities does not quite match the power of promql, and you start to find yourself needed them to debug issues. At that point, the self hosted LGTM stack starts to make sense, however, it involves a lot more education for end users in both integration (a little less now Otel is popular) and querying/building dashboards etc, but also running it yourself. The grafana cloud platform is more attractive though.

Helm diff will show you a similar diff to terraform. Running Helmfile in CD isn't a bad move, its really simple, and its a pattern that is easy to grok by any engineer. I think this is still a valid approach in a simple setup, its what some people call "CD OPS". It's a push model instead of pull, and there are downsides, but its not the end of the world.

Ultimately, at scale, i think gitops tooling like Flux and ArgoCD are some of the nicest patterns. Especially Flux's support for OCI artifacts as a source of truth. However then you will venture into the realm of kustomize, and much more complex tooling and concepts, which is not always worth doing.

I have used both, but find Argo can be unnecessarily complex, and focuses solely as git as a source of truth for your k8s resources. The image updater can even write back to git to reflect version numbers etc, which is arguably an anti-pattern (git is not a database). However, the UI is excellent and is very powerful, and if your just getting started in the gitops space, its very intuitive.

I feel like the weaveworks team (who created flux) have encountered the problem of using git as a source of truth at scale. They let you specify other sources such as S3 and OCI containers, this gives you a lot more power to build custom, powerful workflows.

This means that you define your k8s resources (kustomizations definitions defining k8s resources, and flux resources) in git, but build, lint and test them in a CI/CD pipeline and publish them as a container. Then you can just tag that container with the cluster name or environment and treat your k8s resources like you would code. You can observe this with the flux ui too.

I think people get too hung up on the git part of gitops. All infrastructure should be defined in a version control system, and follow a sane CI process, but the way your cluster pulls that state to enforce it should be any source that is a reflection of that versioned code in SCM.