> AIUI, people are even experimenting with offloading the KV cache itself to storage, which may unlock this batching capability even beyond physical RAM limits as contexts grow.

Especially this point. Any reason that this idea was considered bad? Is it due to the speed difference between the GPU VRAM to the RAM?

PS. mentioning amp because i used to use it and I pay directly for token. I topped up 5 usd so I will be going to use it and see how far can it take me. But my impression so far is even when model subsidization is done, those open source models are quite viable alternatives.

While I agree with the sentiment, I think that might have been initially driven by older models being nerfed and/or newer ones were better at token/$. And there is this notion that those labs don't constraint the model on the first days after its release.

Not to say anything against Zed though. But sublime with one session of claude can help you build your very own customized ide.

"I just wanted you to know that you can't just say the word "AGI" and expect anything to happen.

- Michael Scott: I didn't say it. I declared it

Another small points: reduce your expenses. Basically plan for the worst in terms of budgeting. Widen your search space. There are other younger markets in global south you can also approach.

Theres this expression i couldn't exactly translate to English. It goes along the lines of loosen up your body (literal translation), but it's more about yielding to the flow and less about physically doing so.