In the meantime I've benchmarked Orthrus some more and got some quite promising results. So I'd be glad if my prediction that it may take some time until it lands in llama.cpp turns out to be wrong.

I didn't run the full benchmark with the demo code, just picked up a single prompt from it. The prompt is about 1300 token, the response is about 3200 token.

Baseline: 44.8 t/s With Orthrus: 164.6 t/s

Note: Don't use the `use_diffusion_mode=` config flag in their example to collect a baseline. Something about how the fallback to "normal" makes it grind to a crawl.

At the moment not even MTP is merged into llama.cpp, so I wouldn't quite hold my breath for it.

Their approach is essentially a speculative decoding approach where multiple tokens are predicted at once and then verified. Therefore getting more tokens to be created at a speed that is closer to the prompt processing speed.

It seems to be special because their approach yields the exact same output distribution as the base model and it only takes a negligable amount of additional memory.

The main catch is that if your prompt processing speed is already bad, it will not help you all that much.

For example, the M-series Macs (up to M4) have a relative high generation speed compared to their prompt processing speed. That means they will not benefit as much (if at all). With the M5 the prompt processing speed has increased 4x, so those can expect to see a good uplift.

One particular thing, unrelated to the linguistic argument itself, stood out to me. In the PCA visualisation, we can see that some sequences of layers have particularly tight and stationary clusters. Incidentally, those are also exactly the layers that the previous RYS post identified as most useful to repeat to improve perfomance on the probes.

I wonder, if that correlation could be used to identify good candidates for repeating layers.

Comments URL: https://news.ycombinator.com/item?id=47591834

Points: 15

# Comments: 7

Good thinking on asking Claude to walk you through on who to contact. I had no idea how to contact anyone related to PyPI, so I started by shooting an email to the maintainers and posting it on Hacker News.

While I'm not part of the security community, I think everyone who finds something like this, should be able to report it. There is no point in gatekeeping the reporting of serious security vulnerabilities.

So the fact that 1.82.7 and then 1.82.8 were released within an hour of each other was highly suspicious.

I was just setting up a new project, and things behaved weirdly. My laptop ran out of RAM, it looked like a forkbomb was running.

I've investigated, and found that a base64 encoded blob has been added to proxy_server.py.

It writes and decodes another file which it then runs.

I'm in the process of reporting this upstream, but wanted to give everyone here a headsup.

It is also reported in this issue: https://github.com/BerriAI/litellm/issues/24512

Comments URL: https://news.ycombinator.com/item?id=47501426

Points: 938

# Comments: 500

After building a proof of concept, we decided that we will only continue Teams integration if anyone is going to pay serious money for it.

Unrelatedly, I came across a recomendation for David Burns "Feeling Good" here on hackernews a couple of years ago.

Reading it with my interest in stoicism in mind, I honestly found it to be probably the best modern day handbook to actually adopting the stoic mindset - without ever mentioning it.

As far as I understand stoicism, it is all about seeing things as they are, and understanding that the only thing that we really control is our reaction / interpretation of events. And the CBT approach that is explained in Feeling Good/Feeling Great is exactly how you do this.

With this perspective Marcus Aurelius Meditations suddenly make a lot more sense. They are his therapy homework.