Hacker News: brancz

Continuous Nvidia CUDA Profiling in Production

brancz — Wed, 22 Oct 2025 14:05:54 +0000

Article URL: https://www.polarsignals.com/blog/posts/2025/10/22/gpu-profiling

Comments URL: https://news.ycombinator.com/item?id=45669377

Points: 98

# Comments: 10

New comment by brancz in "Strobelight: A profiling service built on open source technology"

brancz — Fri, 07 Mar 2025 20:15:43 +0000

Thank you!

Overhead ultimately depends on the frequency, it defaults to 19hz per core, at which it’s less than 1%, which is tried and tested with all sorts of super heavy python, JVM, rust, etc. workloads. Since it’s per core it tends to be plenty of stacks to build statistical significance quickly. The profiler is essentially a thread-per-core model, which certainly helps for perf.

The offset approach has evolved a bit, it’s mixed with some disassembling today, with that combination it’s rock solid. It is dependent on the engine, and in the case of python only support cpython today.

New comment by brancz in "Strobelight: A profiling service built on open source technology"

brancz — Fri, 07 Mar 2025 18:23:14 +0000

Already been done:

1) native unwinding: https://www.polarsignals.com/blog/posts/2022/11/29/dwarf-bas...

2) python: https://www.polarsignals.com/blog/posts/2023/10/04/profiling...

Both available as part of the Parca open source project.

https://www.parca.dev/

(Disclaimer I work on Parca and am the founder of Polar Signals)

New comment by brancz in "Strobelight: A profiling service built on open source technology"

brancz — Fri, 07 Mar 2025 15:38:50 +0000

We’re working hard to bring a lot of Strobelight to everyone through Parca[0] as OSS and Polar Signals[1] as the commercial version. Some parts already exists much to come this year! :)

[0] https://www.parca.dev/

[1] https://www.polarsignals.com/

(Disclaimer: founder of polar signals)

New comment by brancz in "Rust: Doubling Throughput with Continuous Profiling and Optimization"

brancz — Fri, 14 Feb 2025 14:05:36 +0000

The point of the first one is that you can create snapshots from within the product where profiling data isn't forever. This is so you can use the pprof.me link in a GitHub issue, PR, or elsewhere and trust that the data never goes away even if the original data went out of retention. We actually originally built pprof.me out of frustration that users of Prometheus (several of us are Prometheus maintainers) at best submitted screenshots of profiling data when all we wanted was an easy way to explore it.

I agree that neither of these are terribly complicated features, but as far as I know no other product on the market actually has this combination. (yes, you can export data from most systems and use a different visualization tool but the point of products is to provide a single integrated package)

(disclaimer: Founder of the company that offers the product featured in this case study.)

New comment by brancz in "Rust: Doubling Throughput with Continuous Profiling and Optimization"

brancz — Fri, 14 Feb 2025 13:42:13 +0000

Keep an eye out on our blog we're working on some interesting things in this area!

New comment by brancz in "Rust: Doubling Throughput with Continuous Profiling and Optimization"

brancz — Fri, 14 Feb 2025 13:41:26 +0000

I mentioned this on another thread as well, but the point isn't that perf can't catch something like this, but it's that having continuous profiling set up makes it way easier to make profiling data an everyday part of your development process, and ultimately nothing behaves quite like production. This allows things that gradually sneak into the codebase to be easily spotted since you don't have to go through the whole shebang of getting representative production profiling data over time, because you just always have it available. Continuous profiling also allows you to spot intermittent things easier and so on.

(disclaimer: I'm the founder of the product showcased in this case study.)

New comment by brancz in "Rust: Doubling Throughput with Continuous Profiling and Optimization"

brancz — Fri, 14 Feb 2025 12:24:43 +0000

Nobody is saying that a regular profiling tool can't detect it. However, it's one of those things that if you don't profile it regularly then these are easy things to miss. With a continuous profiler set up you skip everything regarding collection of the right data (and you can see aggregates across time which in the second example of the blog post was important because the memory allocations aren't necessarily seen in a 10s profile). It makes the barrier for including profiling data in your everyday software engineering way easier.

You can capture memory or CPU metrics with top as well, and that's useful, but it's not the same thing as a full-blown metrics system eg. Prometheus.

New comment by brancz in "Rust: Doubling Throughput with Continuous Profiling and Optimization"

brancz — Fri, 14 Feb 2025 10:59:44 +0000

Prometheus gives you CPU/memory total metrics, the profiler used in the article gets you much higher resolution: down to the line number of your source code.

If you're looking to optimize your project I would recommend using a profiler rather than metrics that just tell you the total.

(disclaimer: I'm the founder of the company that offers the product shown in the blog post, but I also happen to be a Prometheus maintainer)

New comment by brancz in "Rust: Doubling Throughput with Continuous Profiling and Optimization"

brancz — Fri, 14 Feb 2025 10:07:54 +0000

Thank you for the feedback! Quickly worked with the S2 team to get the screenshot from the change added (it's just enabling the hardware acceleration feature in the sha2 crate)!

New comment by brancz in "Show HN: Perforator – cluster-wide profiling tool for large data centers"

brancz — Fri, 31 Jan 2025 16:46:18 +0000

I'm familiar with the paper, but it doesn't improve the situation in terms of LBR availability on cloud providers, does it?

New comment by brancz in "Show HN: Perforator – cluster-wide profiling tool for large data centers"

brancz — Fri, 31 Jan 2025 16:18:31 +0000

If I'm understanding correctly, this is collecting LBR data through hardware support for PGO/AutoFDO, right?

Cloudy Forecast: How Predictable Is Communication Latency in the Cloud?

brancz — Wed, 29 Jan 2025 13:46:56 +0000

Article URL: https://arxiv.org/abs/2309.13169

Comments URL: https://news.ycombinator.com/item?id=42864819

Points: 1

# Comments: 0

Kubezonnet: Monitor Cross-Zone Network Traffic in Kubernetes

brancz — Thu, 09 Jan 2025 15:30:17 +0000

Article URL: https://www.polarsignals.com/blog/posts/2025/01/09/introducing-kubezonnet

Comments URL: https://news.ycombinator.com/item?id=42646593

Points: 4

# Comments: 0

Introducing S2

brancz — Sat, 21 Dec 2024 15:11:19 +0000

Article URL: https://s2.dev/blog/intro

Comments URL: https://news.ycombinator.com/item?id=42480105

Points: 372

# Comments: 195

New comment by brancz in "A Walk with LuaJIT"

brancz — Sat, 23 Nov 2024 13:32:39 +0000

There isn’t, because the indented way to use Parca is to profile production and always-on.

However, we wouldn’t be against adding a mode like this!

FWIW both the server and the agent are single statically linked binaries so while it’s a bit more set up it’s not terribly difficult either[1].

[1] https://www.parca.dev/docs/quickstart/

New comment by brancz in "A Walk with LuaJIT"

brancz — Thu, 21 Nov 2024 21:59:17 +0000

Glad you liked it! Yeah, we worked with a customer who really needs this badly and has done some unspeakable things to get by until now.

New comment by brancz in "A Walk with LuaJIT"

brancz — Thu, 21 Nov 2024 21:03:11 +0000

Thanks for submitting! We know HN has a sweet spot for LuaJIT, so we figured it would eventually end up here.

Quick summary: this post dives into the gory details of how we implemented an eBPF based profiler for LuaJIT.

Let us know if you have any questions on this, we’ll keep an eye out on comments!

New comment by brancz in "(Mostly) Deterministic Simulation Testing in Go"

brancz — Wed, 29 May 2024 12:19:26 +0000

Thanks for posting!

We missed this being submitted yesterday, so let us know if you have any questions, we'll be sure to watch this thread!

New comment by brancz in "How to found a company in Germany: 14 "easy" steps and lots of pain"

brancz — Sun, 14 Apr 2024 14:11:51 +0000

There certainly are, but I’d recommend making the first priority to look for who is actually going to be your most valuable business partner, their location in my opinion doesn’t really play any role in it. And in my experience a VC with conviction in your business will find a way to invest.