Comments URL: https://news.ycombinator.com/item?id=49445884

Points: 89

# Comments: 46

Comments URL: https://news.ycombinator.com/item?id=48940261

Points: 1

# Comments: 0

Regarding tamper proof audit logs not much is missing. Cryptographic hashes instead of XXH3, a commit hash chain and signed commits. Actually, I think that's a great addition with minimal changes needed.

What you can audit currently is "who changed what" for instance.

Furthermore, a (versioned, as always) path summary keeps track of all distinct paths, which is a key ingredient for the optional secondary indexes, which can index paths or paths and content among other stuff as simply indexing fields. Optionally you can also add indexes to speed up aggregate queries (which are basically column projections).

Furthermore, the whole storage can self-validate through checksums stored in parent pages up to the root as in ZFS for instance.

DeweyIDs can be optionally stored for each node, which can speed up the comparison of subtrees between revisions (in order to detect simply which changesets belong to a certain subtree without having to traverse ancestor chains). They lend themselves well for compression.

We even have importers which can identify based on heuristics minimal edit operations to import existing revisions of XML or JSON documents and to commit these with hopefully a minimal or close to minimal set of update operations, but it depends heavily on the data.

The core idea behind SirixDB is, that history is a first-class citizen. Every commit stores a lightweight, queryable revision. You can query any point in time, even individual nodes (for instance JSON values), diff arbitrary revisions, and efficiently track how data evolved without replaying events.

Unlike traditional event stores, historical states do not need to be reconstructed by replaying events nor do we have to think about projections. Revisions are directly queryable.

A simple example:

Jan 1: Record "Price = $100, valid from Jan 1". Stored on Jan 1 (transaction time).

Jan 20: Discover price was actually $95 on Jan 1. Commit correction.

After correction, you can ask across both axes:

- "What did we THINK the price was on Jan 16?" -> $100 (Transaction time)

- "What WAS the price on Jan 1?" -> $95 (Valid time)

I've worked on this in my spare time since 2013, following its academic precursor (Idefix/Treetank) at the University of Konstanz. The architecture relies on an append-only physical log and a persistent copy-on-write page trie.

A high level view of the architecture:

Physical Log (append-only, sequential writes)

  ┌────────────────────────────────────────────────────────────────────────┐
  │ [R1:Root] [R1:P1] [R1:P2] [R2:Root] [R2:P1'] [R3:Root] [R3:P2'] ...    │
  └────────────────────────────────────────────────────────────────────────┘
       t=0      t=1     t=2      t=3      t=4       t=5       t=6    → time

  [Rev 1]          [Rev 2]          [Rev 3]
     │                │                │
     ▼                ▼                ▼
  [Root₁]          [Root₂]          [Root₃]
   │   │            │   │            │   │
   │   └─────────┐  │   └────────┐   │   └─────────┐
   ▼             ▼  ▼            ▼   ▼             ▼
  ┌──────┐       ┌──────┐       ┌──────┐       ┌──────┐
  │  P1  │       │  P2  │       │ P1'  │       │ P2'  │
  └──────┘       └──────┘       └──────┘       └──────┘
   Rev 1         Rev 1+2        Rev 2+3         Rev 3
                 (shared)       (shared)

Back in 2019, and even in 2023, SirixDB was very slow due to GC pressure. Unlike most other document stores, SirixDB stores fine-grained nodes, and I came to realize that an on-heap (JVM) representation made up of lots of small objects simply didn't make sense. I measured it with async-profiler — with some help from Andrei Pangin himself — and the result was that the poor throughput was due to the sheer amount of allocations which scaled almost linearly with the number of open transactions.

Working a full-time software engineering job, I lacked the energy for a massive spare-time rewrite. About a year ago, I started experimenting with AI. It turned out to be ideal for automating the tedious, repetitive parts of migrating the storage layer to Java's Foreign Function & Memory API, storing pages completely off-heap.

Looking further ahead, the append-only, immutable-page design maps naturally onto object storage like S3 and distributed logs like Kafka for a cloud version, and initial prototypes already exist. Maybe that becomes a commercial service one day, but for now, I'm just thrilled to see these core design principles finally proven out.There's an interactive demo, documentation, and the code is on GitHub. I'd love feedback and am happy to answer questions!

kind regards

Johannes

[1] https://sirix.io | https://github.com/sirixdb/sirix

[2] https://sirix.io/docs/architecture.html

[3] https://demo.sirix.io

[4] https://sirix.io/docs/

[5] http://brackit.io

Comments URL: https://news.ycombinator.com/item?id=48922631

Points: 13

# Comments: 6

Comments URL: https://news.ycombinator.com/item?id=48909919

Points: 1

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=48472941

Points: 5

# Comments: 1

Comments URL: https://news.ycombinator.com/item?id=48321526

Points: 3

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=46799224

Points: 2

# Comments: 0

That said, whenever I find some time, I can work on my vision much more efficiently (mostly as a product owner + architect in one person rather than writing everything "by hand"). So, I of course wonder if our jobs are safe in the future. I think you'll always have to heavily guide the agents and stop them immediately whenever they're about to get haywire and thus you have to have the skills of a senior software engineer, but on the other hand I'm sure that small teams of senior engineers can be much more efficient than before. So, either it's that you'll need less software engineers at some point, or if it's rather that you can deliver products faster with more ideas implemented or simply that new ideas can be explored much more efficiently as before => more small startups?). I really don't know...

Comments URL: https://news.ycombinator.com/item?id=46765666

Points: 2

# Comments: 3

Comments URL: https://news.ycombinator.com/item?id=46721131

Points: 2

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=46586424

Points: 1

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=46421733

Points: 3

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=46335524

Points: 1

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=46098325

Points: 2

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=45978268

Points: 1

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=45833515

Points: 3

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=45758196

Points: 2

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=45719674

Points: 3

# Comments: 0