Events, time-series data, user sessions, click, logs, IOT sensor readings, etc. generate a lot of data over time. While on-disk storage works well for Postgres’ rowstore, it’s a poor choice for fast growing data that requires analysis. To avoid the scale limit of on-disk storage, Hydra separates compute and storage. Also, we're not charging separately for bandwidth since it's been factored into the overall plan price.

While storage volume can be a good proxy, many people see the limits of Postgres with a complex join and filtering on relatively small data volumes. With decoupled columnstore and serverless processing, Hydra can be used in big (and small data) use-cases. Company size is a little less relevant since medium and large-scale companies have use-cases where efficient 'small data' is needed too.

Ideally, you can easily switch over to Hydra. Or Hydra can work as a fast, external analytics database too. It's Postgres-native so no changes are needed to use it in a traditional architecture if you wanted to.

Feel free to DM me on X (@JoeSciarrino) or email founders@ so we can coordinate on the SJC region.

One of the downsides of serverless is that it can be difficult to predict the overall monthly cost when the granularity of billing (per invocation, memory usage, or execution time) is complex. For developers this might be totally fine (even preferred), but we think that giving a single, predictable price: Hydra $100 / month is better for businesses to plan around.

Usage caps per plan are purely soft limits so users don't actually encounter them. Yes, we want people to upgrade to higher plans. In the words of Maya Angelou "Be careful when a naked person offers you a shirt" - meaning, we believe these are the best prices we can offer today to build a sustainable project on. That said, I appreciate your point about our # of users limit. If we removed that limit would you try out Hydra?

We initiall set the rowstore as default, but people wouldn't create columnstore tables and were confused on why performance wasn't improving. So, figured this was cleaner, but you always have the option to switch the default table type back.

Traditionally, this was unfeasible: Postgres is a rowstore database that’s 1000X slower at analytical processing than a columnstore database.

(A quick refresher for anyone interested: A rowstore means table rows are stored sequentially, making it efficient at inserting / updating a record, but inefficient at filtering and aggregating data. At most businesses, analytical reporting scans large volumes of events, traces, time-series data. As the volume grows, the inefficiency of the rowstore compounds: i.e. it's not scalable for analytics. In contrast, a columnstore stores all the values of each column in sequence.)

For decades, it was a requirement for businesses to manage these differences between the row and columnstore’s relative strengths, by maintaining two separate systems. This led to large gaps in both functionality and syntax, and background knowledge of engineers. For example, here are the gaps between Redshift (a popular columnstore) and Postgres (rowstore) features: (https://docs.aws.amazon.com/redshift/latest/dg/c_unsupported...).

We think there’s a better, simpler way: unify the rowstore and columnstore – keep the data in one place, stop the costs and hassle of managing an external analytics database. With Hydra, events, traces, time-series data, user sessions, clickstream, IOT telemetry, etc. are now accessible as a columnstore right alongside my standard rowstore tables.

Our solution: Hydra separates compute from storage to bring the analytics columnstore with serverless processing and automatic caching to your postgres database.

The term "serverless" can be a bit confusing, because a server always exists, but it means compute is ephemeral and spun up and down automatically. The database automatically provisions and isolates dedicated compute resources for each query process. Serverless is different from managed compute, where the user explicitly chooses to allocate and scale CPU and memory continuously, and potentially overpay during idle time.

How is serverless useful? It's important that every analytics query has its own resources per process. The major hurdles with running analytics on Postgres is 1) Rowstore performance 2) Resource contention. #2 is very often overlooked - but in practice, when analytics queries are run they tend to hog resources (RAM and CPU) from Postgres transactional work. So, a slightly expensive analytics query has the ability to slow down the entire database: that's why serverless is important: it guarantees the expensive queries are isolated and run on dedicated database resources per process.

why is hydra so fast at analytics? (https://tinyurl.com/hydraDBMS) 1) columnstore by default 2) metadata for efficient file-skipping and retrieval 3) parallel, vectorized execution 4) automatic caching

what’s the killer feature? hydra can quickly join columnstore tables with standard row tables within postgres with direct sql.

example: “segment events as a table.” Instead of dumping segment event data into a s3 bucket or external analytics database, use hydra to store and join events (clicks, signups, purchases) with user profile data within postgres. know your users in realtime: “what events predict churn?” or “which user will likely convert?” is immediately actionable.

Thanks for reading! We would love to hear your feedback and if you'd like to try Hydra now, we offer a $300 credit and 14-days free per account. We're excited to see how bringing the columnstore and rowstore side-by-side can help your project.

Comments URL: https://news.ycombinator.com/item?id=43937852

Points: 60

# Comments: 33

Features: Serverless Processing

- Parallel, vectorized excution

- Compute Autoscale

Bottomless Storage

- 10X data compression

- Automatic caching

- zero-copy snapshots & forks

hydra: serverless realtime analytics on postgres. by separating compute and storage, hydra enables compute-isolated analytics and bottomless storage. it is designed for low latency applications built on time series and event data.

set up is simple.

> pip install hydra-cli

$ hydra

traditionally, there's been 2 core problems with running analytics on postgres.

- slow performance: aggregate queries can take minutes to return results from large data sets.

- resource contention: expensive queries hog Postgres’ ram / cpu resources and impair transactional performance.

here’s how hydra solves these problems:

- fast: hydra returns analytics queries 400X faster than standard postgres. hydra uses duckdb to perform isolated serverless processing on these tables in postgres. In fact, hydra is faster than most specialized analytics databases.

- isolated serverless processing: there is no impact on postgres ram / cpu resources.

“ok, but why use postgres?” is a common question. it’s true, there are many specialized analytics databases, but they create their own frictions and costs:

- moving and transforming data (ETL) between postgres, s3 bucket, and separate analytics db isn’t cheap. pipelines speed determines how stale the analytics are. hydra side-steps the latency and costs of data pipelines entirely with full support for inserts and updates on columnar files in the analytics schema.

- many use-cases don’t justify a heavy setup. from our time working at heroku, we saw many transactional apps that just need a couple high level aggregates and a few complex analytical queries. hardly olap and not really htap - just apps that need a boost.

Comments URL: https://news.ycombinator.com/item?id=43343870

Points: 2

# Comments: 1

Our collaboration with Hydra revolves around pg_duckdb, an open-source (MIT licensed) program that embeds DuckDB’s state-of-the-art analytics engine and features within Postgres. pg_duckdb is meant for developing high-performance applications and analytics with any new or existing Postgres database. We’ve observed software engineers increasingly embedding powerful analytics directly into their applications. These applications tend to require both greater access to disparate data sources and sub-second response times. We believe pg_duckdb will serve these use-cases nicely by overcoming Postgres’ known limitations in analytical processing. ... continues in article