Hacker News: charles_irl

New comment by charles_irl in "Cutting inference cold starts by 40x with LP, FUSE, C/R, and CUDA-checkpoint"

charles_irl — Mon, 18 May 2026 22:55:22 +0000

To clarify: we do content-based hashing, and when we say "shared bytes aren’t guaranteed to be in the exact same container image layer", what we mean is that

FROM some/image RUN pip install torch==2.7.1

and

FROM another/image RUN pip install torch==2.7.1

will produce images with very high overlap in contents, which will be shared by a content-based cache, but those images' final layers are disjoint from the perspective of a layerwise cache.

New comment by charles_irl in "Cutting inference cold starts by 40x with LP, FUSE, C/R, and CUDA-checkpoint"

charles_irl — Mon, 18 May 2026 20:34:47 +0000

You're absolutely right!

New comment by charles_irl in "Cutting inference cold starts by 40x with LP, FUSE, C/R, and CUDA-checkpoint"

charles_irl — Mon, 18 May 2026 20:08:51 +0000

Yep! That should start in ten seconds or so -- about a second per gigabyte of weights, plus a second to start the container and a few seconds to load the memory snapshot.

There are a few limitations with snapshotting, e.g. it generally fails when using multiple GPUs, which we document here: https://modal.com/docs/guide/memory-snapshots.

New comment by charles_irl in "Cutting inference cold starts by 40x with LP, FUSE, C/R, and CUDA-checkpoint"

charles_irl — Mon, 18 May 2026 19:05:38 +0000

Cutting latencies by 40x! Unfortunately couldn't fit the whole title in the character limit :<

Cutting inference cold starts by 40x with LP, FUSE, C/R, and CUDA-checkpoint

charles_irl — Mon, 18 May 2026 17:56:26 +0000

Article URL: https://modal.com/blog/truly-serverless-gpus

Comments URL: https://news.ycombinator.com/item?id=48183038

Points: 65

# Comments: 15

How to Achieve Serverless GPUs

charles_irl — Tue, 12 May 2026 16:23:16 +0000

Article URL: https://modal.com/blog/truly-serverless-gpus

Comments URL: https://news.ycombinator.com/item?id=48110438

Points: 8

# Comments: 0

New comment by charles_irl in "Keeping 20k GPUs healthy"

charles_irl — Thu, 22 Jan 2026 21:03:58 +0000

> Ed Zitron also called out the business model of GPU-as-a-service middleman companies like modal deeply unsustainable, and I also don't see how they can make a profit if they are only reselling public clouds.

You got a link for that? I work on Modal and would be interested in seeing the argument!

We think building a proper software layer for multitenant demand aggregation on top of the public clouds is sufficient value-add to be a sustainable business (cf DBRX and Snowflake).

New comment by charles_irl in "Three types of LLM workloads and how to serve them"

charles_irl — Thu, 22 Jan 2026 03:46:09 +0000

Sorry to lead with a bunch of jargon! Wanted to make it obvious that we'd give concrete recommendations instead of palaver.

The technical terms there are later explained and diagrammed, and the recommendations derived from something close to first principles (e.g. roofline analysis).

New comment by charles_irl in "Three types of LLM workloads and how to serve them"

charles_irl — Thu, 22 Jan 2026 03:14:19 +0000

oof ty, willfix

Three types of LLM workloads and how to serve them

charles_irl — Wed, 21 Jan 2026 16:15:06 +0000

Article URL: https://modal.com/llm-almanac/workloads

Comments URL: https://news.ycombinator.com/item?id=46707708

Points: 75

# Comments: 5

Host overhead is killing your inference efficiency

charles_irl — Tue, 18 Nov 2025 19:37:50 +0000

Article URL: https://modal.com/blog/host-overhead-inference-efficiency

Comments URL: https://news.ycombinator.com/item?id=45970964

Points: 3

# Comments: 0

New comment by charles_irl in "Quantized Float Exposed"

charles_irl — Mon, 03 Nov 2025 20:09:38 +0000

Inspired by https://float.exposed, which was on the front page recently, I put together this visualizer for lower precision/quantized floating point numbers -- specifically, all of the formats in the Open Compute Project's Microscaling Formats Spec (https://www.opencompute.org/documents/ocp-microscaling-forma...).