I built a system once which had (this is Rust) a Rayon worker thread pool of 4 threads and a Tokio async pool of 2 (multithreaded runtime). On a system of 6 vCPU. This ended up working fine. Tokio was not starved so handled network requests at low latency.

One difference is DuckDB is a pure network client. If one of its async threads is starved it is not the end of the world (e.g. k8s does not kill your pod for failure of replying to health checks).

That is fantastic. Last I checked models capable of running on commodity (anything below a dedicated GPU rack) hardware were very lackluster.

    server:
        @echo "Deploying machine (with ssh-agent forwarding): '$(MACHINE)'"
        NIX_SSHOPTS="-A" \
        nix run nixpkgs#nixos-rebuild -- switch \
        --flake .#$(MACHINE) \
        --target-host $(MACHINE).$(DOMAIN) \
        --build-host $(MACHINE).$(DOMAIN) \
        --no-reexec \
        --verbose \
        --sudo

This uses ssh agent forwarding, and then sudo via PAM. That allows for passwordless sudo. Building (well, activating) without sudo is pretty involved last I checked, I could not get it to work.

Says it right there actually:

> It’s recommended to not set this limit too low in order to avoid hanging on operations requiring spawn_blocking.

So a total like 6 - reasonable for a web backend - would be way too low.

For genuine CPU-bound work, submitting to a Rayon worker pool is the way to go. It solved a runtime starvation issue for us at work, spawn_blocking did not work.

The reason for all this is spawn_blocking having a very large underlying thread pool, in the hundreds. That is okay if you assume work will yield those threads and mostly sleep/wait. It is not okay if the work never yields, like pure data crunching. (Go solves this by forcefully preempting loops, no such thing in Rust without a language runtime)

Our solution shape was: multi-threaded Tokio (2 threads), then give the rest of available_concurrency to a Rayon thread pool. If you grant 6 vCPU you should see a thread pool of 4, and a maximum CPU consumption of about 400%, as the Tokio threads sit mostly idle (under low load single-threaded runtime should also suffice).

You inject the thread pool using an Arc.

Then, when work comes in, just spawn Tokio tasks liberally (cheap) and submit to the thread pool. Rayon will internally queue and limit concurrency and parallelism to 4 (this is the important bit compared to spawn_blocking: no way your system can hog all 6 threads with non-yielding work and starve Tokio runtime threads).

We use one-shot channels to submit results back, they are designed for exactly this. The tx aka sender end is sync, as there is never a wait (cannot block). The rx aka receiver side is async and can be awaited normally on the async side. This is a cheap operation, similar to Go.

Optionally you can reach for semaphores to also limit I/O concurrency. You probably want to do this for more control and avoiding resource exhaustion loudly (that is, not silently accidentally peg thousands of FDs, database connections, …).

It ended up working beautifully for our purposes and relatively simply. No lifetime woes, Arc solves those. Oneshot channels just transfer ownership etc.

Perhaps this is what TFA talks about, I have not read it.

One caveat: to reach all the above conclusions and designs, we had help from some genuine Rust experts. As much as I dislike Go, it "just works" there even if one writes naive code.

I also run self-hosted Wireguard. Initially on a Debian box, nowadays it is integrated into my router (admittedly, this is closed source). For around 6 years at this point.

The whole thing could not be easier and simpler. It has never randomly broken on me. It is fast. It is free. There is no middle man, no vendor.

I never understood the popularity of Tailscale, though that is on me. I'm sure it is a great product, I just never tried it, do not seem the target audience.

What confuses me is the often accompanying, sometimes aggressive anti-selfhosting stance in these sorts of threads. I do not see this in other topics, e.g. someone mentioning they run Jellyfin isn't met with "why not Plex?". Where does that come from? We are on HackerNews, not ProductShillNews, aren't we? I guess self hosting Wireguard is too boring to warrant any further discussion? The VPN equivalent of a Toyota Corolla.

> I suspect rather than hire less people we will just produce more code changes.

Why? Towards what end? Code changes are output, not outcome. It also needs to be connected to someone willing to pay you hard cash. That is the hard part, a race to the bottom, and the reason I also believe there will be downwards pressure on salaries and even employment.

They do, but you need far fewer or none of the original workers whose full-time job this sort of stuff was.

Raw math does not matter, but what you do with it. Similarly, you could earn a (modest) living knowing nothing but raw HTML, JavaScript and a bit of browser tech not too long ago. That is no longer possible.

Programming and software engineering will be devalued. These occupations won't disappear overnight, but you will see compensation and growth stagnate until equilibrium is reached again. Currently, supply outstrips demand, and I do think it is structural, not just hype.

I'm certainly not creative enough, but I currently do not see demand picking up sufficiently; Gen Z is bearish on social media, VR was a bust, blockchain was a bust, software has already penetrated almost all walks of live and lines of work. There is no next big thing (Internet, ...) on the horizon, to unlock the next order of magnitude of demand. There is certainly more work to do still, but it very suddenly does not require the same headcount, but something like 5%-30% less. Lots of the remaining work will be around integrating LLMs into existing software, which does not sound exciting either.

That is problematic for two reasons: it might be a large type, so copying might be expensive. Second, more likely, it might violate invariants in your domain. For a rate limiter, this might mean accidentally copying around some internal state like a mutex, which then exists n times instead of 1 time, which can represent a problem (e.g. if you want to internally limit whole-app concurrency toward Redis).

> that step didn't add any security.

It is a decision that’s part of the entire process. A branch of many in the decision tree. Other branches are deciding which characters to type for the password; ASCII characters can be as little as 1 bit apart. Deciding between left and right is also 1 bit apart.

I think it boils down to what people commonly understand to be publicly knowable information versus understood-to-be-secret information.

One example: I self-host my password manager at pw.example.com/some-secret-path/. That extra path adds as much to security as a randomly picked username in HTTP Basic Auth: arguably none. Yet, it is as impossible for attackers to enumerate and find that path as it is with passwords.

The difference is that the path leaks easier. It’s not generally understood to be a secret. Yet I argue it helps security. (Example: leaking the domain name through certificate transparency logs AND even, say, user credentials means an attack is still unsuccessful; a strictly necessary piece of the puzzle is missing).

> No Silver Bullet

As an industry, we do not know how to measure productivity. AI coding also does not increase reliability with how things are going. Same with simplicity, it's the opposite; we're adding obscene complexity, in the name of shipping features (the latter of which is not productivity).

In some areas I can see how AI doubles "productivity" (whatever that means!), but I do not see a 10x on the horizon.

> Kernighan's Law

Still holds! AI is amazing at debugging, but the vast majority of existing code is still human-written; so it'll have an easy time doing so, as indeed AI can be "twice as smart" as those human authors (in reality it's more like "twice as persistent/patient/knowledgeable/good at tool use/...").

Debugging fully AI-generated code with the same AI will fall into the same trap, subject to this law.

(As an aside, I do wonder how things will go once we're out of "use AI to understand human-generated content", to "use AI to understand AI-generated content"; it will probably work worse)

> just ask AI to rewrite the code

This is a terrible idea, unless perhaps there is an existing, exhaustive test harness. I'm sure people will go for this option, but I am convinced it will usually be the wrong approach (as it is today).

> Dijkstra on the foolishness of programming in natural language

So why are we not seeing repos of just natural language? Just raw prompt Markdown files? To generate computer code on-the-fly, perhaps even in any programming language we desire? And for the sake of it, assume LLMs could regenerate everything instantly at will.

For two reasons. The prompts would either need to raise to a level of precision as to be indistinguishable from a formal specification. And indeed, because complexity does become "exponentially harder"; inaccuracies inherent to human languages would compound. We need to persist results in formal languages still. It remains the ultimate arbiter. We're now just (much) better at generating large amounts of it.

> Lehman’s Law

This reminds me of a recent article [0]. Let AI run loose without genuine effort to curtail complexity and (with current tools and models) the project will need to be thrown out before long. It is a self-defeating strategy.

I think of this as the Peter principle applied to AI: it will happily keep generating more and more output, until it's "promoted" past its competence. At which point an LLM + tooling can no longer make sense of its own prior outputs. Advancements such as longer context windows just inflate the numbers (more understanding, but also more generating, ...).

The question is, will the market care? If software today goes wrong in 3% of cases, and with wide-spread AI use it'll be, say, 7%, will people care? Or will we just keep chugging along, happy with all the new, more featureful, but more faulty software? After all, we know about the Peter principle, but it's unavoidable and we're just happy to keep on.

> Jevons Paradox

My understanding is the exact opposite. We might well see a further proliferation of information technologies, into remaining sectors which have not yet been (economically) accessible.

0: https://lalitm.com/post/building-syntaqlite-ai/

The upside is there's no single entity receiving all your queries. The downside is there's no encryption (IIRC root servers do not support it), so your ISP sees your queries (but they don't receive them).