Comments URL: https://news.ycombinator.com/item?id=45887797

Points: 17

# Comments: 1

It's a pretty neat algorithm from a paper in 2019 for the application "to index k-mers of DNA samples or q-grams from text documents". You can take a collection of bloom filters built for documents and then combine them together to have a single filter that will tell you which docs it maps to. Like an inverted index meets a bloom filter.

I'm using it in a totally different domain for an upcoming release in InfluxDB (time series database).

There's also code online here: https://github.com/bingmann/cobs

I'm excited to start doing some experimentation with Vortex to see how it can improve our products.

Great stuff, congrats to Will and team!

Comments URL: https://news.ycombinator.com/item?id=44814290

Points: 19

# Comments: 1

The 2.x to 3.x move is, admittedly, much harder. This is because of the language Flux. We haven't been able to bring that over to 3.x in a way that makes it useful. We actually built a bridge for it in our cloud offering, but our experience is that the performance isn't good enough to be acceptable for customers wanting to upgrade. If they want to make the move, adopting SQL or InfluxQL is likely the only path.

We'll continue to develop 3.x and we'll build more migration tooling over time. I think we can build specialized tooling to help Flux users migrate over to 3.x with query translation tools, but there are more features we need to land in 3.x to enable that first.

We're committed to the technology stack (Apache Arrow & DataFusion) and the 3.x line. We have no plans for another major release. I'll be happy if we end up releasing 3.56.2 8 years from now.

Then start over again to clean things out. It's not flawless, but it is surprising what it'll remember from a while back in the conversation.

I've been meaning to pick up some of the more automated tooling and editors, but for the phase of the project I'm in right now, it's unnecessary and the web UI or the Claude app are good enough for what I'm doing.

Right now I'm swapping between Gemini and Opus depending on the task. Gemini's 1M token context window is really unbeatable.

But the quality of what Opus 4 produces is really good.

edit: forgot to mention that this is all for Rust based work on InfluxDB 3, a fairly large and complex codebase. YMMV

Comments URL: https://news.ycombinator.com/item?id=43996189

Points: 13

# Comments: 2

We wrote and contributed the Rust object store crate used in this stack and by many others to the ASF.

This release is based on a "diskless" architecture that uses object storage for all durability. With DataFusion it has a columnar, vectorized, standards compliant SQL query engine. We also built support for InfluxQL on top of it.

The other big thing we brought in is an embedded Python VM using PyO3 and Python Build Standalone. This makes it possible to do data collection, ETL, monitoring, alerting, and all kinds of tasks inside the database at the point of collection.

Happy to answer any questions about the big project, what's next or anything time series related.

Comments URL: https://news.ycombinator.com/item?id=43693209

Points: 5

# Comments: 1

The InfluxDB part of it is more narrowly scoped than previous versions, but the Telegraf and Kapacitor parts are much more feature rich than those previous products.

Can you say more about your use case?

The commercial offering is the historical time series DB along with a bunch of other features around high availability, read replication, fine grained security, and the compaction engine which enables longer range queries and row level deletes.

I think Scylla had most of their DB in the open and then a small slice of Enterprise functionality (although I'm not super familiar with their product line).

Ideally, we'd have many open source users and even our commercial customers would use the open source in addition to the commercial offering.

But ultimately, it's about finding a sustainable business model that keeps more software coming. We have a preference for permissive open source over source available. In my view, we may as well create freemium rather than source available.

With this version of InfluxDB, we've been able to invest heavily into Apache projects that lie at the core of it: Arrow, DataFusion, Parquet, and the object store crate, which we developed and donated to the ASF.

We'd like to continue that work because we think that a highly performant, modular, vectorized query engine (i.e. DataFusion) should be a free commodity that's widely available and widely contributed to.

I'm guessing there will be questions about the 72 hour limit. There are two things we're looking at:

First, we're considering giving a free tier for at home and hobbyist usage of Enterprise, which doesn't have this limitation. So this would be kind of like what Tailscale does giving a free usage plan for their commercial software.

Second, for Core, the open source build, we're working on an update that will let it query any 72 hour window of historical data. Right now it doesn't evict data, it all still exists on disk or object storage as Parquet files, but we remove the metadata information from RAM to keep things optimized for the most recent 72 hours.

When the update is done, you'll be able to write and query for any period of time. But an individual query will be limited to a 72 hour time range. This is a service protection mechanism because of how the data is organized.

A file gets created for every 10 minute block of time for each table. So 72 hours is 432 files, which is a lot of GET requests to S3 for a single query. We don't want to increase the range because of that. Multiple queries combining a longer range, or accessing the data from third-party clients is all still possible.

In Enterprise, our commercial product, we have a compactor that collapses these files into larger time blocks that also creates an index that the query engine can use.

Doing it this way was a deliberate choice so that we could have a permissively license open source project separate from the commercial product. If we put the compactor into the open, we'd have to put it under a source available license to limit usage so that we can still sell the database.

Our hope is that there's still an audience of users that will find Core useful on its own, even without any commercial relationship with us. It's not a full historical TSDB, but it's not intended to be. It's meant to be a recent data engine that can collect, process, monitor, ship, and store data paired with a fast analytical query engine against the recent buffer (or recently persisted buffer).

Happy to answer any followup questions about this or the release generally.

So Core isn't intended to be a full historical TSDB. It's more like a data collector, processing engine, data shipper and recent data buffer/DB.

For a full historical TSDB, that's the product we sell. Keeping the two separate gives us the ability to have real open source vs. combining them and requiring a different license that lets us do freemium.

We'll likely have a freemium tier for the commercial product (Enterprise), but that's separate from the open source project.

It's a constraint that we could relax over time, but for now we wanted to limit the scope so we can focus on the recent data. We're also considering a free tier of Enterprise for at-home use cases (i.e. non-commercial hobbyist).

As for EOL on previous versions, we don't have anything planned at the moment. We're partnered with AWS on their hosted versions of InfluxDB 2.x OSS so I expect that to continue for quite some time.