Hahahahhahahaha

Incredible.

Do you have any tips for Gemma 4 31B in particular? I quite like the model but I feel like I'm underutilizing my rented GPU hard due to skill issues. Throughput should be way higher than this, right?

---

Gemma 4 31B NVFP4, vLLM 0.29, single B200 (modal), FlashInfer, fp8 KV, prefix caching, 32k ctx

Workload: ~6k-token shared prefix (~98% cache hit) + short input, ~250 tokens out, ~250 seqs running. Get ~5-6k output tok/s, flat from 128 to 384 concurrency.

Spec decode (DFlash, n-gram) didn't give a good boost which is annoying because I feel like this task should be easy for a draft model to predict.

Individual request latency doesn't matter I just need as much throughput as possible. Its only active for a few hours when I need it.

"give me this again without jargon invented this session at high density

and with a couple (maybe more or less) simple useful ascii diagrams underneath each design"

The context is that I was discussing an experimental new idea for my video game review analysis product.

Designs 1,2, and 3 were horrible: the model even suggested a rejection after the word soup so it would have been pointless to waste my fleeting time on Earth reading it.

Otherwise, I generally really enjoyed using fable for bouncing ideas. It was an absolute joy to have this thing provide useful criticism, analyse sample data, and create prototypes so that I could elevate my understanding of the problem without stepping down from a pure intuition/design headspace.

But I don't consider the purely model written code usable for a feature this important. I'll probably scrap it entirely and start from scratch with newfound understanding.

Try asking them to make useful diagrams for some stuff in a codebase, out of the box without excessive hand holding they don't make good choices about what's worth communicating and how to do it.

You see this in their pointless frontend copy all the time too.

But even fable has the annoying tendency to invent new jargon and produce an incomprehensible soup of text.

I have been using the --watch flag to let my agent play with the notebook as I use it already.

But sometimes I want to do something too specific or high fidelity and it's just easier to get the clanker to write typescript and make a webpage/components.

However my use of it has decreased a little with how easily I can conjure disposable frontends with agents to explore one off things.

I'm using the VS Code extension over SSH.

I love Fable for many things including coding but I cannot justify basing any internal AI tools or LLM powered products on top of Anthropic's offerings.

> preventing misuse.

Imagine not being able to read the tokens you are paying for.