I found with using various tricks I can get the zstd parquet to be up to 10x (or more) smaller than an equivalent Postgres table - but obviously the exact compression ratios will depend on the kind of data you have and how well your Postgres table is normalized.

  UPDATE my_table
  SET x = file1.x,
      y = file2.y
  FROM 'first_file.csv' file1
  LEFT JOIN 's3://my_bucket/second_file.parquet' file2
    ON file1.id = file2.id
  WHERE mytable.id = file1.id;

You can also write a skill that CC can re-use if you're analyzing a lot of similar data sets with minor variance.

It has connectors for Postgres & other stores, so I find it faster to connect to a Postgres instance, pull all of the data from a table (even if the table is like 50GB - if you have 30 cores on the machine it will pull from Postgres using 30 cores in parallel, so it will only take a minute or two) - and then any analytical queries on the data are 10+ times faster in DuckDB over native Postgres (GROUP BY, regexp_replace, count(distinct...) etc).

I do a lot of experiments with regexes, and if you get used to the RE2 syntax that DuckDB uses, you can see up to 10-100x uplift in terms of speed compared to Postgres on things like regexp_matches(), regexp_extract(), etc (depending on query/table/machine specifics). It has quite powerful scripting with custom Macros, fixes a lot of annoyances of SQL for me compared to Postgres.

I think if you have access to a machine with a lot of RAM / cores and a beefy data set, then it's basically like a RAMdisk version of Snowflake running locally on your machine.

(and of course the fact that it makes it convenient to read CSV/parquet, read/write from S3, etc) - it's a very ergonomic tool.

It definitely increases some types of productivity (Opus one-shot a visualization that would have likely taken me at least a day to write before, for work) - although I would have never written this visualization before LLMs (because the effort was not worth it). So I guess it's Jevons Paradox in action somewhat.

In order to observe the productivity increases you need a good scale where the productivity would really matter (the same way that when a benchmark is saturated, like the AIME, it stops telling us anything useful about model improvement)

If you had more like 200GB ram you might be able to run something like MiniMax M2.1 to get last-gen performance at something resembling usable speed - but it's still a far cry from codex on high.

The author uses MIT Living Wage numbers to argue that should be the new "poverty" benchmark - an absurd proposition. Those might be reasonable middle class numbers. He also implies that the benchmark historically represented what is now covered under that $140K calculation - also false; it took ~$9000 in 1966 to cover a "basic standard of living" for a family of 4 with 1 earner; inflation adjusted, that's around $90,000 today. If you add in SS/Medicare taxes (3% then, 15% today), that puts you at ~$100K-105K.

Using the same MIT Living Wage numbers and taking Essex-Princeton NJ as the area (roughly what the author used), you end up with $99,922 as the living wage for a single earner, 4 person household - almost exactly the same as the household back in 1966.

Were there more jobs in 1966 that paid $9000/year versus jobs that pay $100K today? That's the real story you're looking for.

The author uses MIT Living Wage numbers to argue that should be the new "poverty" benchmark - an absurd proposition. Those might be reasonable middle class numbers. He also implies that the benchmark historically represented what is now covered under that $140K calculation - also false; it took ~$9000 in 1966 to cover a "basic standard of living" for a family of 4 with 1 earner; inflation adjusted, that's around $90,000 today. If you add in SS/Medicare taxes (3% then, 15% today), that puts you at ~$100K-105K.

Using the same MIT Living Wage numbers and taking Essex-Princeton NJ as the area (roughly what the author used), you end up with $99,922 as the living wage for a single earner, 4 person household - almost exactly the same as the household back in 1966.

Were there more jobs in 1966 that paid $9000/year versus jobs that pay $100K today? That's the real story you're looking for.

Therefore the adjustment should probably be based on a different number reflecting the actual consumption of households near the poverty line (food would probably be higher than it is in the CPI currently, as one example)

This is more akin to manager-level view of the code (who need developers to go and look at the "deterministic" instructions); the abstraction is a lot lot more leaky than high->low level languages.

Subcultures? Some are at least trying to (i.e. rationalists), though imperfectly and with side-effects.

And I find that even the auto-generated stuff tends to go up at least a bit in terms of level of abstraction than staring at the code itself, and helps you more like a "sparknotes" version of the code, so that when you dig in yourself you have an outline/roadmap.

I would expect that this will cause certain programs to see more demand than the creators anticipated for (extrapolating previous trends), which might require changes in the programs (i.e. more people apply for benefits than expected, benefits / application might have to be cut, etc).

And in some ways there's a Cantillon effect (though traditionally associated with proximity to the "money printer", but here the proximity is to the LLM-enablement; in that those who use the LLMs first can get the benefit before the rules are changed).

From my understanding the AlphaZero based engines (i.e. neural nets) do not really benefit from having a human in the loop.

A Slav that was born and grew up in China would be Caucasian but culturally Japanese.

Accusations of racism should stay limited to disparate treatment based on skin color or race exclusively and not extend into the cultural domain. Cultural preferences are not racist, full stop.