Comments URL: https://news.ycombinator.com/item?id=48339459

Points: 4

# Comments: 0

[1] https://www.cbs.nl/en-gb/news/2025/50/ever-more-gas-free-hom... [2] https://www.abnamro.nl/en/personal/specially-for/preferred-b...

The work-around solution we used at Trustly (a company I co-founded), is a component named `allas` that a colleague of mine at that time, Marko Tikkaja, created to solve our problems, that massively reduced the load on our servers. Marko has open sourced and published this work here: https://github.com/johto/allas

Basically, `allas` opens up a single connection to PostgreSQL, on which it LISTEN on all the channels it needs to listen on. Then clients connect to `allas` over the PostgreSQL protocol, so it's basically faking a PostgreSQL server, and when clients do LISTEN on a channel with allas, allas will then LISTEN on that channel on the real PostgreSQL server on the single connection it needs. Thanks to `allas` being implemented in Go, using Go's efficient goroutines for concurrency, it efficiently scales with lots and lots of connections. I'm not a Go-expert myself, but I've understood Go is quite well suited for this type of application.

This component is still being used at Trustly, and is battle-tested and production grade.

That said, it would of course be much better to avoid the need for a separate component, and fix the scalability issues in core PostgreSQL, so that's what I'm currently working on.

In general, a single-queue design doesn’t make throughput collapse when you add more parallelism; it just gives you a fixed ceiling. With a well-designed queue, throughput goes up with concurrency, then flattens when the serialized section (the queue) saturates, maybe sagging a bit from context switching.

If instead you see performance severely degrade as you add workers, that typically means there’s an additional problem beyond “we have one queue” — things like broadcast wakeups (“every event wakes every listener”), global scans on each event, or other O(N) work per operation. That’s a very different, and more serious, scalability bug than simply relying on a single queue.

> Even if you have 20 database connections making 20 transactions in parallel, all of them need to wait for their turn to lock the notification queue, add their notification, and unlock the queue again. This creates a bottleneck especially in high-throughput databases.

We're currently working hard on optimizing LISTEN/NOTIFY: https://www.postgresql.org/message-id/flat/6899c044-4a82-49b...

If you have any experiences of actual workload where you are currently experiencing performance/scalability problems, I would be interested in hearing from you, to better understand the actual workload. In some workloads, you might only listen to a single channel. For such single-channel workloads, the current implementation seems hard to tweak further, given the semantics and in-commit-order guarantees. However, for multi-channel workloads, we could do a lot better, which is what the linked patch is about. The main problem with the current implementation for multi-channel workloads, is that we currently signal and wake all listening backends (a backend is the PostgreSQL processes your client is connected to), even if they are not interested in the specific channels being notified in the current commit. This means that if you have 100 connections open in which each connect client has made a LISTEN on a different channel, then when someone does a NOTIFY on one of those channels, instead of just signaling the backend that listen on that channel, all 100 backends will be signaled. For multi-channel workloads, this could mean an enormous extra cost coming from the context-switching due to the signaling.

I would greatly appreciate if you could please reply to this comment and share your different workloads when you've had problems with LISTEN/NOTIFY, to better understand approximately how many listening backends you had, and how many channels you had, and the mix of volume on such channels. Anything that could help us do better realistic simulations of such workloads, to improve the benchmark tests we're working on. Thank you.

This is only true if those notifications are different; if they are identical, such as in the same the notification is to alert listeners some table has new data (for cache invalidation), they are sent out as one notification only. See source code comment in async.c:

     *   Duplicate notifications from the same transaction are sent out as one
     *   notification only. This is done to save work when for example a trigger
     *   on a 2 million row table fires a notification for each row that has been
     *   changed. If the application needs to receive every single notification
     *   that has been sent, it can easily add some unique string into the extra
     *   payload parameter.

    postgres=# SELECT
        (pg_identify_object(s.classid,s.objid,s.objsubid)).*,
        pg_catalog.pg_get_acl(s.classid,s.objid,s.objsubid) AS acl
    FROM pg_catalog.pg_shdepend AS s
    JOIN pg_catalog.pg_database AS d
        ON d.datname = current_database() AND
        d.oid = s.dbid
    JOIN pg_catalog.pg_authid AS a
        ON a.oid = s.refobjid AND
        s.refclassid = 'pg_authid'::regclass
    WHERE s.deptype = 'a';
    -[ RECORD 1 ]-----------------------------------------
    type     | table
    schema   | public
    name     | testtab
    identity | public.testtab
    acl      | {postgres=arwdDxtm/postgres,foo=r/postgres}

Do people here need pg_ownerships and/or pg_privileges?

[1] https://www.postgresql.org/message-id/flat/bbe7d1cb-0435-4ee...

https://github.com/graphile/worker

    // Line 535 in src/main.ts
    client.query('LISTEN "jobs:insert"; LISTEN "worker:migrate";')

We're now working on improving the scalability of LISTEN/NOTIFY in PostgreSQL, and to guide that work, I'd like to better understand how it's used (or was used) in real-world systems. What works well? What doesn't?

The current implementation has some known scalability bottlenecks:

1. Thundering Herd Problem: A NOTIFY wakes up all listening backends in the current database, even those not listening on the notified channel. This is inefficient when many listeners are each listening to their own channels (e.g. in job queues).

2. Commit Lock Contention: NOTIFY operations are serialized behind a heavyweight lock at transaction commit. This can become a bottleneck when many transactions send notifications in parallel.

If you've used LISTEN/NOTIFY in production, I'd love to hear:

- What is/was your use case?

- Does each client listen on its own channel, or do they share channels?

- How many listening backend processes?

- How many NOTIFYs in parallel?

- Are you sending payloads? If so, how large?

- What worked well for you? What didn't?

- Did you hit any scalability limits?

Feedback much appreciated, thanks!

/Joel

Comments URL: https://news.ycombinator.com/item?id=44821373

Points: 2

# Comments: 4

To do Passkey-verification server-side, I had to implement a pure-SQL/PLpgSQL CBOR parser, out of fear that a C-implementation could crash the PostgreSQL server: https://github.com/truthly/pg-cbor