1. Great Python support. Piping something from a structured data catalog into Python is trivial, and so is persisting results. With materialization, you never need to recompute something in Python twice if you don’t want to — you can store it in your data catalog forever.

Also, you can request anything Python package you want, and even have different Python versions and packages in different workflow steps.

2. Catalog integration. Safely make changes and run experiments in branches.

3. Efficient caching and data re-use. We do a ton of tricks behind to scenes to avoid recomputing or rescanning things that have already been done, and pass data between steps with Arrow zero copy tables. This means your DAGs run a lot faster because the amount of time spent shuffling bytes around is minimal.

RE: workflow orchestrators. You can use the Bauplan SDK to query, launch jobs and get results from within your existing platform, we don’t want to replace entirely if it’s doesn’t fit for you, just to augment.

RE: DuckDB and Polars. It literally uses DuckDB under the hood but with two huge upgrades: one, we plug into your data catalog for really efficient scanning even on massive data lake houses, before it hits the DuckDB step. Two, we do efficient data caching. Query results and intermediate scans and stuff can be reused across runs.

More details here: https://www.bauplanlabs.com/blog/blending-duckdb-and-iceberg...

As for Polars, you can use Polars itself within your Python models easily by specifying it in a pip decorator. We install all requested packages within Python modules.

This one, although it’s a custom orchestration system, not Kubernetes. (there are some similarities but our system is really optimized for data workloads)

We manage Iceberg for easy data versioning, take care of data caching and Python modules, etc., and you just write some Python and SQL and exec it over your data catalog without having to worry about Docker and all infra stuff.

I wrote a bit on what the efficient SQL half takes care of for you here: https://www.bauplanlabs.com/blog/blending-duckdb-and-iceberg...

> In the end I kind of understand it as similar to sqlmesh, but with a "BYO compute" approach? So where sqlmesh wants to run on a Data Warehouse platform that provides compute, and only really supports Iceberg via Trino, bauplan is focused solely on Iceberg and defining/providing your own compute resources?

Philosophically, yes. In practice so far we manage the machines in separate AWS accounts _for_ the customers, in a sort of hybrid approach, but the idea is not dissimilar.

> Should I understand therefore that this is only usable with an account from bauplanlabs.com ?

Yep. We’d help you get started and use our demo team. Send jacopo.tagliabue@bauplanlabs.com an email

RE: pricing. Good question. Early startup stage bespoke at the moment. Contact your friendly neighborhood Bauplan founder to learn more :)

Comments URL: https://news.ycombinator.com/item?id=42727305

Points: 6

# Comments: 0

Salespeople harangue you for calls because it's objective fact that it works to bring more dollars in, and the idea that they say some magic words and then the customer suddenly wants to buy is childish. They identify and address needs and pain points.

Comments URL: https://news.ycombinator.com/item?id=42251200

Points: 1

# Comments: 2

But more importantly, in digging in -- to me, that's a big part that's missing in leveling up the next generation -- like hey, there's a stack trace, let's go look at the lines of code in our source libraries and think about them instead of flailing around randomly like most people seem to.

I wonder if it would be slightly less sad if we all had our own decentralized crawlers that simply functioned as extensions of ourselves.

If I want to relax a bit, no more than 5mg of an edible. I've taken way more at various bits in life, letting it become waaay too much of a habit, and I think it causes little but trouble. It often resulted in a bad loop of increased tolerance, being really tired the next day, slamming caffeine to over-compensate, having more anxiety to deal with, repeat...

CBD, cliche as it is to say, is actually pretty underrated as an in-between. With 1mg THC, 20mg CBD, it's very soothing without being impairing.

I think with smoking and vaping it's exceedingly easy to consume way too much. I'd recommend not more than a couple puffs, to be honest. Weed is way too strong now. My kingdom for a return to the 60's situation of weak weed and strong acid.

I suspect it’s more about the culture than the numbers. You say on one hand, oh I’d probably stay for 3% more, yet, you don’t see the point to earn another 1%. Your salary is pretty good man. Meanwhile they are burning $250K every month into smoke

10% is not realistic. That’s the whole employee options pool. I mean imagine if at your current job, someone worked there for a year before you, then walked away with more than your entire current equity grant, never to be heard from again. That’s what you’re describing.

But for some people, direct access to an AWS account and license to build as they please is intoxicating.

But anyway, as founding engineer you get to set the systems, culture, language etc. maybe some people don’t want the responsibility but for others it’s an opportunity to build things out in our own image and learn a lot.