I thought the principal consequence of these KV cache optimisations was letting you run more simultaneous inferences on the same model with the same memory. It doesn’t let you store more model. In some sense that puts local LLM usage at a further disadvantage to inference done in a hyperscaler’s data center.

It seems like a fatal dilution of focus to have to worry about the design and logistics of a fancy dumb terminal widget when you also have to get the software/AI/app integration stuff right.

Just make an app with text and voice interaction. Accept that the thing in our pockets with a screen and an internet connection is going to be a smartphone. You will not build an own-hardware moat with these weird little bits of e-waste.

I would give you the benefit of the doubt that it might just be code shyness or perfectionism about something in its early stages, but it looks like the last codec you developed (“HALIC”) is still only available as Windows binaries after a year.

I struggle to see an upside to withholding source code in a world awash with performant open source media codecs.

I’d have thought mainframe dates were mostly binary coded decimal in EBCDIC and already futureproofed during the y2k mania.