Comments URL: https://news.ycombinator.com/item?id=40362574

Points: 1

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=40059185

Points: 2

# Comments: 0

Vitess (Handles Wire Protocol)

go-mysql-server (GMS, Implementation of MySQL featureset)

Dolt GMS interface

Storage Engine

DoltgreSQL, right now, shares the storage engine, interface, and a portion of GMS. This is primarily to get something up and running so that we're able to further iterate on a "working" product (i.e. something we can set up testing for). I envision that GMS will split into three, with a MySQL & PostgreSQL core interacting with a base SQL core. Dolt's GMS interface would then adapt to the base SQL core interface. The storage engine is more-or-less set in stone.

There are two key points regarding performance that I'd like to mention, and that's that Dolt uses higher-than-expected disk, and we also use more memory than MySQL by comparison. The disk usage is due to our optimization of speed at the sacrifice of disk accretion via temporary storage, which we've decided is a fair trade off considering disk is very cheap (and we're working on making this tunable so users can decide on speed vs disk efficiency). Memory usage is a bit more complex, and I'm not the employee to comment too much about it, but both of these issues are being worked on to reduce their impact. With that in mind, our performance is comparable to MySQL as long as the machine limits are not being reached, however I'd expect us to be a bit slower once those limits are reached, simply due to the extra complexity that we're managing.

Lastly, trust is something that can only be built over time. In 10 years, I'm sure that there will be no doubt of our stability, and at that time I can see Dolt and DoltgreSQL becoming the de-facto databases used for relational storage. Of course, I may be a bit biased :)

Trust is a

There are products that can do something similar for schemas, but DoltgreSQL (and Dolt, the production-ready DB based around MySQL rather than PostgreSQL) applies this to both schemas and data. This is unique to DoltgreSQL and Dolt, and what sets us apart from all other products. One example I like to use that I think really captures this power, is the ability to query data from any commit. We use Git's model of version control, so your database has a commit history and a working set. The working set can be viewed as the "current" commit. When you run queries, you implicitly select the working set, however we expose the ability to explicitly set which commit you're targeting. This means you can do things like run a join query across old and current data!

These situations, and more, and not possible via migration tools, backup tools, or any other kind of tools. This is unique to DoltgreSQL and Dolt, by fully embracing data versioning. This is but one example, and there are many, many more.

Recovery is a valid use-case for version control, but fully applying it to data opens up possibilities that haven't really been explored before.

https://github.com/dolthub/dolt

In fact, we've got a blog post about Dolt and Wordpress.

https://www.dolthub.com/blog/2023-08-04-wordpress-on-dolt/

https://docs.dolthub.com/concepts/dolt/rdbms/replication

Regarding any differences from mainstream Postgres, you can look to how we've handled Dolt, which is production-ready. It targets MySQL, just as DoltgreSQL targets PostgreSQL, and it recently achieved 99.99% correctness according to a set of roughly 6 million tests (https://www.dolthub.com/blog/2023-10-11-four-9s-correctness/). This test is not a definitive stance that we are exactly 99.99% the same as MySQL, but it's a good general guide to how we approach our compatibility, and how serious we are in that regard.

Sadly though, the full versioning capabilities would not work as an extension to Postgres. We looked into it before we settled on our current approach. I talk a bit about it in the blog post as well. To truly allow versioning in the same capacity that Git does for source code, it required us to either fork Postgres and spend years reimplementing all of the work that we've done in Dolt just to get to where we are today, or choose the path that gets something out quickly, and allows us to have the very conversation that we're having right now.

That was actually the very reason for deciding to host an announcement that we're working on it. Many people have said that they'd like Dolt but for Postgres, however they've not said whether they need the Postgres binary specifically, the ecosystem, the syntax/wire protocol, etc. This announcement gives us the opportunity to receive that feedback.

Dolt (and eventually DoltgreSQL) handles everything. Branching, merging, diffing, cherry-picking, commits, and more. Working with both the schema and data. On top of that, we also have DoltHub (https://www.dolthub.com/) that's analogous to GitHub, and DoltLab (https://www.dolthub.com/#doltlab) that's analogous to GitLab. We are targeting the entire ecosystem from the bottom up.

https://www.dolthub.com/blog/2022-07-11-dolt-case-studies/

Either way, with DoltgreSQL now in development, we will now have solutions for both sides! (Other popular databases with their own syntax, such as Oracle and Microsoft, are not planned for the future at this time)

The versioning features are an important distinction though. Dolt (production ready, MySQL protocol) and DoltgreSQL (pre-alpha, PostgreSQL protocol) are built specifically to address the lack of versioning support in databases, and gaining these versioning features is as easy as swapping out the database you are using for Dolt and DoltgreSQL (once it's finished). MySQL and PostgreSQL are written using C/C++, while Dolt and DoltgreSQL are using Go, so there is no shared code. The storage format is implemented using prolly trees (https://docs.dolthub.com/architecture/storage-engine/prolly-...), which are based on merkle trees (used by Git and Bitcoin), so there is no overlap with any existing database solutions.

So it looks like VMDS is a rather unique take on versioning data, where data is viewed as objects. Dolt and DoltgreSQL use the table and row paradigms that the majority of modern relational databases use. Dolt is designed to be a drop-in replacement for MySQL, and DoltgreSQL for PostgreSQL, so that already determines the interface. Regarding VMDS' versioning functionality, I see that is supports merging and conflict resolution at the transaction layer, but I'm not seeing anything similar to the concept of branches, which is something that Dolt and DoltgreSQL supports.

Overall, for modern use cases, it doesn't look like they overlap much. VMDS seems focused more on the spatial case while representing data as objects, while Dolt and DoltgreSQL are traditional relational databases that support versioning all aspects of a relational database.

Performance optimizations are tackled a bit differently than correctness ones. For the most part, we'll try to use metrics to find weak points in the execution graph and optimize those, but we won't go so far as to try and model the internal performance behavior. In part because our storage format is so different that we'll have different performance characteristics by necessity.

We don't have a public roadmap for Doltgres just yet, but we're hoping to put one out quite soon! We have a lot of low-hanging roadblocks that we want to take care of before we can get a better look at the overall time frame. We should definitely have one up by the end of the month, but I don't want to commit to a time before that.