I'd used it in a past life to build a Kubernetes scheduler [2] and tackle some cluster management problems.

[1] https://www.youtube.com/watch?v=lxiCHRFNgno [2] https://www.usenix.org/system/files/osdi20-suresh.pdf

Feldera tries to be row- and column-oriented in the respective parts that matter. E.g. our LSM trees only store the set of columns that are needed, and we need to be able to pick up individual rows from within those columns for the different operators.

I don't think we've converged on the best design yet here though. We're constantly experimenting with different layouts to see what performs best based on customer workloads.

Some folks pointed out that no one should design a SQL schema like this and I agree. We deal with large enterprise customers, and don't control the schemas that come our way. Trust me, we often ask customers if they have any leeway with changing their SQL and their hands are often tied. We're a query engine, so have to be able to ingest data from existing data sources (warehouse, lakehouse, kafka, etc.), so we have to be able to work with existing schemas.

So what then follows is a big part of the value we add: which is, take your hideous SQL schema and queries, warts and all, run it on Feldera, and you'll get fully incremental execution at low latency and low cost.

700 isn't even the worst number that's come our way. A hyperscale prospect asked about supporting 4000 column schemas. I don't know what's in that table either. :)

For anyone else, if you want to try out Feldera and IVM for feature-engineering (it gives you perfect offline-online parity), you can start here: https://docs.feldera.com/use_cases/fraud_detection/

Comments URL: https://news.ycombinator.com/item?id=46645570

Points: 3

# Comments: 0

If the goal is to maintain views over graphs and performance/scale matters, consider Feldera. We see folks use it for its ability to incrementally maintain recursive SQL views (disclaimer: I work there).

* "Slope beats y-intercept." The best computer scientists and engineers I've ever worked with and/or mentored embodied this principle more than anything else.

* It can be tempting to over-optimize for short-term milestones (e.g., an important admissions exam, the next job or promotion), but there is a significant compounding value to knowledge accumulation and truly learning your craft well. Read and learn as much as you can, all the time, even if it isn't immediately necessary or useful.

Comments URL: https://news.ycombinator.com/item?id=44767888

Points: 35

# Comments: 8