Hacker News: geoHeil

New comment by geoHeil in "Incremental – A library for incremental computations"

geoHeil — Tue, 21 Jul 2026 07:04:23 +0000

I maintain a similar library more focused on data engineering needs: https://docs.metaxy.io/ maybe it is useful for more people.

New comment by geoHeil in "Show HN: ELO-Ranking Analytics Engines"

geoHeil — Sun, 31 Aug 2025 05:46:03 +0000

Benchmarks are like single match scorelines: interesting, but rarely comparable. We built a public ELO score leaderboard for analytics engines that treats every public benchmark (TPC-DS, TPC-H, SSB, vendor blogs, community posts - even the messy ones) as a “match.”

Why ELO? Because comparing databases is more like ranking football clubs over a season than timing a single sprint. Creating a relative robust and relative score is what counts.

Show HN: ELO-Ranking Analytics Engines

geoHeil — Sun, 31 Aug 2025 05:43:50 +0000

Article URL: https://data-inconsistencies.datajourney.expert

Comments URL: https://news.ycombinator.com/item?id=45080649

Points: 6

# Comments: 1

New comment by geoHeil in "LLM-in-a-Box: A Templated, Self-Hostable Framework for Generative AI"

geoHeil — Sun, 20 Jul 2025 20:23:06 +0000

We are currently preparing a submission for JOSE and are looking for feedback but also more people who are interested in joining in and refining the template

New comment by geoHeil in "LLM-in-a-Box: A Templated, Self-Hostable Framework for Generative AI"

geoHeil — Sun, 20 Jul 2025 20:22:51 +0000

Together with www.linkedin.com/in/aaron-cul... I have created a template for making LLMs (from different vendors and even self hosted ones) easily accessible to researchers - including advanced document RAG with docling.

LLM-in-a-Box: A Templated, Self-Hostable Framework for Generative AI

geoHeil — Sun, 20 Jul 2025 20:22:51 +0000

Article URL: https://github.com/complexity-science-hub/llm-in-a-box-template

Comments URL: https://news.ycombinator.com/item?id=44628960

Points: 3

# Comments: 3

New comment by geoHeil in "Show HN: An AI tutors that teaches AI"

geoHeil — Tue, 10 Jun 2025 03:19:52 +0000

Nice. What about tracking the learning process and comparing against the peer group/ class?

New comment by geoHeil in "Ask HN: Why don't we have a functional DSL for data+embedding+API pipelines?"

geoHeil — Mon, 14 Apr 2025 15:54:22 +0000

As some others already posted: There is more to copying data than just moving it. It is about observability. A lot of companies have created their own frameworks.

One example I find useful for a lot of usecases is dagster. You can define resources to encapsulate complexity https://docs.dagster.io/guides/build/external-resources in fact with components build on top of custom DSLs https://docs.dagster.io/guides/labs/components

At Magenta/Telekom we are bulding on https://georgheiler.com/event/magenta-data-architecture-25/ - you can follow along with this template here https://github.com/l-mds/local-data-stack/ you may find these examples useful to understand how to use dagster/graph-based data pipeliens @scale

Getting Better at Data Engineering

geoHeil — Mon, 07 Apr 2025 04:47:57 +0000

Article URL: https://georgheiler.com/post/learning-data-engineering

Comments URL: https://news.ycombinator.com/item?id=43607806

Points: 3

# Comments: 0

New comment by geoHeil in "Mix data SaaS workloads (Databricks vs. EMR) to save money by using dagster"

geoHeil — Fri, 03 May 2024 19:03:39 +0000

Did you ever want to find an alternative to cloud data PaaS/SaaS vendor lock-in? In my blog post, I show how a 0-cost switch between local, Databricks and AWS EMR environments can be achieved to save cost by using dagster. Additionally, software engineering best practices around testability, and producing DRY code are enhanced.

Check out my deep dive and let’s discuss the potential impacts on our projects!

Big thanks also to my co-author Hernan.

Mix data SaaS workloads (Databricks vs. EMR) to save money by using dagster

geoHeil — Fri, 03 May 2024 19:03:39 +0000

Article URL: https://georgheiler.com/2024/05/02/cost-efficient-alternative-to-databricks-lock-in/

Comments URL: https://news.ycombinator.com/item?id=40251095

Points: 7

# Comments: 2

New comment by geoHeil in "The new local data stack: Integrating Dagster, dbt and DuckDB"

geoHeil — Fri, 15 Dec 2023 08:04:34 +0000

Interesting why? What were the pain points?

New comment by geoHeil in "The new local data stack: Integrating Dagster, dbt and DuckDB"

geoHeil — Fri, 15 Dec 2023 08:01:23 +0000

If you use bigquery in big lake mode - i.e. with parquet or iceberg or delta files on GCS (or other object store) you can easily pull in the data into duckdb as well

New comment by geoHeil in "The new local data stack: Integrating Dagster, dbt and DuckDB"

geoHeil — Thu, 14 Dec 2023 17:37:35 +0000

Hi,

Georg - one of the authors is here.

What we argue is that:

- for a great software/data engineering/creation experience we recommend such a stack that is only on when needed (when transformations occur)

- for a great data consumption experience we suggest the integration with an established PaaS platform. Not only for the sake of being available (as a serving layer of data to end users) but also for the missing fine-grained RBAC in the proposed transformation layer

New comment by geoHeil in "The new local data stack: Integrating Dagster, dbt and DuckDB"

geoHeil — Thu, 14 Dec 2023 17:33:41 +0000

Well local can mean your laptop. But could mean your local server. However, it can also mean a VM on your cloud provider of choice colocated to the object store.

In such a case the network transfer (for many cases) can be almost irrelevant - at least for up to medium-ish sized datasets.

New comment by geoHeil in "The new local data stack: Integrating Dagster, dbt and DuckDB"

geoHeil — Thu, 14 Dec 2023 17:31:57 +0000

Not to forget https://www.starrocks.io/ or https://risingwave.com/ - see also a recent blog post of a friend: https://juhache.substack.com/p/moving-from-bi-to-data-apps-p...

New comment by geoHeil in "The new local data stack: Integrating Dagster, dbt and DuckDB"

geoHeil — Thu, 14 Dec 2023 17:29:52 +0000

Hi,

Georg - one of the authors is here.

Indeed, we used LLMs/GPT4 for proof reading and enhancing the English language (we are not native speakers).

We were thinking about breaking up the content - but decided that one long post is a better fit.

> > We show a way how you can combine the best of both worlds

In fact, that sentence I wrote by hand : )

New comment by geoHeil in "The new local data stack: Integrating Dagster, dbt and DuckDB"

geoHeil — Thu, 14 Dec 2023 17:26:28 +0000

Hi,

Georg - one of the authors is here.

In fact, this is exactly what we argue: Keep data the consumer experience high a Paas platform (Fabric, BigLake, Databricks, SF, ...) can make a lot of sense, whereas for the best data development/creator experience a high-code solution based on solid software engineering best practices should be the preferred solution - at least in my/our(authors) opinion.

New comment by geoHeil in "The new local data stack: Integrating Dagster, dbt and DuckDB"

geoHeil — Tue, 12 Dec 2023 10:09:19 +0000

We (Aleksandar and Georg) want to share our new blog post on: "Dagster, dbt, DuckDB as new local MDS" here. It re-introduces the local environment to enhance the developer productivity for data pipelines by bringing back software engineering best practices. We suggest that PaaS platforms should become an implementation detail and refine the new local stack with great a data consumer experience by combining the best of both worlds.

Please take a read here: https://georgheiler.com/2023/12/11/dagster-dbt-duckdb-as-new...

We (Georg Heiler and Aleksandar Milicevic) are keen to discuss the proposed new stack with you. Do not hesitate to reach out.

The new local data stack: Integrating Dagster, dbt and DuckDB

geoHeil — Tue, 12 Dec 2023 10:09:19 +0000

Article URL: https://georgheiler.com/2023/12/11/dagster-dbt-duckdb-as-new-local-mds/

Comments URL: https://news.ycombinator.com/item?id=38610892

Points: 144

# Comments: 50