Comments URL: https://news.ycombinator.com/item?id=42095668

Points: 2

# Comments: 0

SQL server is where this breaks though. You'll get yelled by DBAs for bad db practices like storing wide text fields without casting them to varchar(32) or varchar(12), primary keys on strings or no indexes at all, and most importantly taking majority of storage on the db host for tbese raw dumps. SQL Server and any traditional database scales by adding machines, so you end up paying compute costs for your storage.

If you use a shared disk system with decoupled compute scaling from storage, then your system is the way to go. Ideally these days dump your files into a file storage like s3 and slap a table abstraction over it with some catalog and now you have 100x less storage costs and about 5-10x increased compute power with things like duckdb. Happy data engineering!

That's why you build data platforms and name your team accordingly. This is much easier position to defend, where you and your team have a mandate to build tools for other to be efficient with data.

If upstream provides funky logs or jsons where you expect strings, that's for your downstream to worry about. They need the data and they need to chase down the right people in the org to resolve that. Your responsibility should be only to provide a unified access to that external data and ideally some governance around the access like logging and lineage.

Tldr; Open your 'data' mandate too wide and vague and you won't survive as a team. Build data platforms instead.

Comments URL: https://news.ycombinator.com/item?id=41475639

Points: 5

# Comments: 0

Second, there was never really a need to rush to move to another location offering better opportunities. As consequence of the 1990s policies the local capital vanished into the hands of western entities and with it the opportunities worth moving for. The post 2000 capital which moved to the region just found spots with cheap labor to build new factories or logistic centers to keep the German powerhouse running. With an unfair advantage of cheap eastern energy, cheap eastern workers across the border and cheap euro currency as a result of sharing it with the unproductive european south.

It definitely makes things easier to follow, but only for linear, ie. single table, transformations. The moment joins of multiple tables come into the picture things become hairy quick and then you actually start to appreciate the plain old sql which accounts for exactly this and allows you to specify column aliases in the entire cte clause. With this piping you lose scope of the table aliases and then you have to use weird hacks like mangling names of the joined in table in polars.

For single table processing the pipes are nice though. Especially eliminating the need for multiple different keywords for filter based on the order of execution (where, having, qualify (and pre-join filter which is missing)).

A missed opportunity here is the redundant [AGGREGATE sum(x) GROUP BY y]. Unless you need to specify rollups, [AGGREGATE y, sum(x)] is a sufficient syntax for group bys and duckdb folks got it right in the relational api.

If you use embedded duckdb on the client, unless the person goes crazy clicking their mouse at 60 clicks/s, duckdb should handle it fine.

If you run it on the backend and expect concurrent writes, you can buffer the writes in concatenated arrow tables, one per minibatch, and merge to duckdb every say 10 seconds. You'd just need to query both the historical duckdb and realtime arrow tables separately and combine results later.

I agree that having a native support for this so called Lambda architecture would be cool to have natively in duckdb. Especially when drinking fast moving data from a firehose.

It always looked to me as if somebody back then in the database wars tried to word play on each other, one way or another.

https://en.m.wikipedia.org/wiki/QUEL_query_languages

https://github.com/andmarti1424/sc-im

The informal T vs formal V causes confusion in conversation with semi-strangers. Eg. At work you never really know which way to speak to someone at a watercooler. If you choose the informal T you make them your equal. Ehich might be perceived as insulting to them, since they might want to keep a perception of superiority to you for eg. being older, more tenured, etc. Often you'd rather choose not to even engage in a conversation and just keep your thoughts to yourself. Better than ending up in a inferior position when choosing the safe V, or risking insulting someone when using the informal T form.

This felt to me like one of the reasons why English became the dominant language for business over time. Together with simplified morphology (you only need to learn the plural by adding 's' at the end, vs. 5+ other tenses of each word) it just ended up being much easier to pick up and less risky to engage in conversations and therefore higher chances of adoption by non-speakers.

[1] https://en.wikipedia.org/wiki/T%E2%80%93V_distinction

Comments URL: https://news.ycombinator.com/item?id=38747186

Points: 2

# Comments: 0

Please consider removing any implicit network calls like the initial "Checking GitHub for updates...". This itself will prevent people from adoption or even trying it any further. This is similar to gnu parallel's --citation, which, albeit a small thing - will scare many people off.

Consider adding pivot and unpivot operations. Mlr gets it quite right with syntax, but is unusable since it doesn't work in streaming mode and tries to load everything into memory, despite claiming otherwise.

Consider adding basic summing command. Sum is the most common data operation, which could warrant its own special optimized command, instead offloading this to external math processor like lua or python. Even better if this had a group by (-by) and window by (-over) capability. Eg. 'qsv sum col1,col2 -by col3,col4'. Brimdata's zq utility is the only one I know that does this quite right, but is quite clunky to use.

Consider adding a laminate command. Essentially adding a new column with a constant. This probably could be achieved by a join with a file with a single row, but why not make this common operation easier to use.

Consider the option to concatenate csv files with mismatched headers. cat rows or cat columns complains about the mismatch. One of the most common problems with handling csvs is schema evolution. I and many others would appreciate if we could merge similar csvs together easily.

Conversions to and from other standard formats would be appreciated (parquet, ion, fixed width lenghts, avro, etc.). Othe compression formats as well - especially zstd.

It would be nice if the tool enabled embedding outputs of external commands easily. Lua and python builtin support is nice, but probably not sufficient. i'd like to be able to run a jq command on a single column and merge it back as another for example.

Inspiration:

  - csvquote: https://news.ycombinator.com/item?id=31351393
  - teip: https://github.com/greymd/teip

Comments URL: https://news.ycombinator.com/item?id=37189147

Points: 2

# Comments: 2

BUT, leaks memory like crazy.

Despite documentation stating the verbs are fully streaming.

> Fully streaming verbs > These don't retain any state from one record to the next. They are memory-friendly, and they don't wait for end of input to produce their output.

https://miller.readthedocs.io/en/6.7.0/streaming-and-memory/...

They removed the important feature to search only forums, ie. human generated content, and promoted SEO spam to the top instead. Public forums became undiscoverable and people moved to walled gardens of facebook and similar instead.

Then Google killed the search by trying to make it some AI answering robot. Now they ignore what you even ask it and just return to you what they think you'd want.

All that people were asking for was a better search engine and all we got was an inferior version of a chat bot.

Problem of many organizations is they are progressing without a vision and have no idea when progress gets blocked. Since nobody really measures blockage, workers typically drop the thing that got blocked and go chase metrics on something else and don't tell anyone.