Much better off storing this in an embedded graph database and use cypher.

MVCC, Indexes, Strong typing, WAL, changefeeds.

The simplicity of markdown and JSON are deceptive. SQLite is a solid choice, but then you deal with graph-relational impedance mismatch.

KuzuDB folks worked on something called GRainDB in 2022: https://vldb.org/cidrdb/2022/graindb-a-relational-core-graph...

But circa 2023 decided to write their own. Work continues as LadybugDB. One of my long term goals is to find an integration point with DuckDB's table implementation as the "node table". Conversely at some point DuckDB could implement all the join algorithms and LadybugDB's REL table in their code base.

For now, any talk of Graph on DuckDB is limited to DuckPGQ and the more recent entrant DuckGQL (both of which don't touch the storage layer which is the main reason why LadybugDB exists).

Why do we need a source distribution for a well regarded MIT licensed project? Because it's not easy to contribute code to DuckDB if you don't work at DuckLabs. The CI used to take 5 hours for a simple bug fix last I looked (may have improved since I flagged it on social media).

There are two that I'm aware of:

  * Haybarn: https://rusty.today/blog/duckdb-extension-distribution-gap/
  * Pygmy-Goose: https://github.com/Pygmy-Goose/pygmy-goose

  zig build sqlite-benchmark

  Medium (100K nodes)

  +----------------------+-----------+---------+---------+
  | Workload             | LatticeDB | SQLite  | Speedup |
  +----------------------+-----------+---------+---------+
  | 1-hop traversal      | 5.7μs     | 16.1μs  | 2.8x    |
  | 2-hop traversal      | 30.1μs    | 59.4μs  | 2.0x    |
  | 3-hop traversal      | 171.1μs   | 228.8μs | 1.3x    |
  | Variable path (1..5) | 82.2μs    | 5.8ms   | 70.4x   |
  +----------------------+-----------+---------+---------+

Given that on-disk data structures are similar to SQLite, I expect the competition from other "graph on sqlite" projects when they co-opt the techniques in LatticeDB.

https://huggingface.co/datasets/ladybugdb/wikidata-20260401

Couple of corrections:

* LadybugDB has revamped the Kuzu WAL design. It shouldn't be hard to build WAL based replication

* 19ms vs 39us - like the author says these are vastly different systems and the benchmark methodology may not be comparable.

We've mostly focused on query plan optimizations, not so much the micro query operator optimizations.

The 0.20.x end of the month release should have some interesting optimizations.

  * Prepared statements will cache query plans and result vectors. So you don't pay malloc costs
  * SIMD optimizations for filter. More to come in the next release.

These methods could not beat the 480 token result because of the small dataset. But for a sufficiently large corpus with graph reordering and columnar parquet compression, they could be quite competitive.

The parquet files are self explanatory. For knowledge.lbdb.zst, first decompress with zstd and then run a cypher such as:

  lbug> MATCH (a:E)-[t:TRIPLE]->(b:E) RETURN a.name AS subj, t.predicate AS pred, b.name AS obj, t.props AS props, t.scope AS scope;

We don't need to invent new standards. Such interoperable, immutable memory standards already exist: Apache Arrow and Compressed Sparse Rows (CSR). CSR is widely used in scipy, cugraph and columnar graph databases among others. Both on CPUs and GPUs.

icebug-format combines both into a on-disk standard based on Apache Parquet and an in-memory format based on Apache Arrow.

Bindings available in many popular languages including python, typescript and rust.

The package ships with convenience scripts to convert flat tables such as vertex.parquet and edges.parquet to this format in RAM/disk constrained environments.

Sample graphs: https://huggingface.co/datasets/ladybugdb/ldbc-csr/tree/main

Converted from: https://ldbcouncil.org/benchmarks/graphalytics/datasets/

Largest converted graph has 3B edges. License: MIT

Quick Start:

  uv tool install icebug-format
  icebug-format --source-dir wiki-Talk
  lbug -i wiki-Talk-csr/schema.cypher

Comments URL: https://news.ycombinator.com/item?id=49381909

Points: 3

# Comments: 0

> This is particularly problematic if you're trying to introduce fundamental new concepts because you can't change the grammar of Python or C++.

Yet, this is exactly the approach py2many is taking.

Keep it 99% python compatible. Deviate from the python grammar only for new features and where python made a decision incompatible with compiled language backends.

Many users will find transpiling static python to mojo an interesting path.

Updating py2many --mojo to 1.0 to see what breaks.

Also looking forward to translating existing python adt module to mojo Variant in phase 2. I hope the mojo pattern matching proposal looks more like rust (expression) and not like python (statement).

But mojo is a superset, not a subset. So why not use a subset, infer what you need and generate mojo?

I've never seen this question actually presented to the company and discussed in more detail.

You can stick to python and generate mojo or rust or lean.

There is more than one solution to the two language problem.

https://adsharma.github.io/django-fquery/

Your models can be plain old python data classes, declaratively mapped to Django primitives.

  curl  | bash

  fast on the GPU: nx-cugraph
  slow on the CPU: networkx (no arrow, see linked graph-benchmark)

Now that nvidia is becoming a major CPU vendor, they may like the idea too.

https://github.com/Ladybug-Memory/icebug-graphframes-compari...

I thought of datafusion work and the rust implementation as a way to address the delta seen vs the Spark/JVM GraphFrames implementation (author is a major contributor).

Looking forward to more such innovations, which will benefit the ecosystem as a whole. Why would anyone want to use a pure python graph algorithm package unless they're dealing with toy graphs?

https://github.com/timlrx/graph-benchmarks

Not clear if the author or anyone else has an updated version of these benchmarks. Opened an icebug issue on the repo.