to the point where it's really great at coding, but lost its knowledge on how to write well!

if instead opus 5 was a full fresh training run, i don't think it'd be this trash at writing.

really great coding perf, but now the weights are adapted to so much code fine tuning that it writes like a weird engineer that's trying to sound smart.

And I can let you use it on this 6-month window for free because it costs me nothing per user! Your own device does all the processing :D

We all have thousands of buried screenshots, notes, files, bookmarks, saved posts, etc we'll never find again. The only way to make AI understand all of it is to upload everything to the cloud -- privacy nightmare, and way too expensive at scale. And it shouldn't be possible on-device either: small models are too dumb, and phones are too slow for thousands of LLM inference runs.

So I spent a year deeply optimizing every layer of the on-device inference stack to make it possible anyway.

Sentient OS runs a custom multimodal vision LLM on your phone and laptop while they charge overnight. It understands your entire digital life -- every screenshot, note, file, email, bookmark, plus integrations for external services -- with nothing ever leaving your device.

This gives you three things that weren't possible before:

-> Talk to your entire digital life in natural language: "what was that wine I liked?" / "who did I wanna meet next week?" [on-device RAG]. And with MCP, your existing LLM (ChatGPT, Claude, etc.) can talk to your digital life too -- so it actually understands you.

-> Proactive reminders surfaced from your own data: "that tax return in your Downloads is due next week" / "tickets for that concert you screenshotted open tomorrow"

-> Knowledge graphs of your entire digital life: tap any node to find what you buried!

Here's what I had to build to make this possible:

Inference speed:

- KV cache reuse: the system prompt + few-shot examples are identical across all 3,000 analysis calls. I run inference on that prefix once, cache the KV state, and reuse it for every image. Prefill drops to just processing the image itself.

- Thermal-aware scheduling: I throttle the moment iOS reports thermal state > fair. I have all night, so I trade speed for not cooking the device.

- iOS jetsam awareness: iOS kills apps above a specific RAM threshold. I profiled that threshold across different iPhones and push right up to the edge.

Model quality at small size:

- Vision transplant: a 2B Qwen model has terrible vision. I transplanted Qwen 3.5 9B's multimodal projector onto the 2B base. Same architecture family makes this possible.

- Selective quantization on MLX: MLX doesn't support k-quant style mixed precision. I built it manually: less quantization on first/last layers and high-activation layers, more aggressive on the rest.

The alpha processes ~3,000 screenshots entirely on-device on a 6 year old iPhone. Coming to Mac and iPhone!

Previously I researched Apple's neural accelerators: https://www.reddit.com/r/LocalLLaMA/comments/1ohrn20/

And I love OSS! I built https://github.com/theJayTea/WritingTools (2K+ stars, ~30 press features). I'm considering making Sentient OS OSS under AGPL (so no one else can profit off of my work haha).

I think this is one of the coolest consumer usecases to take advantage of on-device LLMs. I'd love to hear what you all think, and happy to answer any questions (I love geeking out about the deep work that's gone into optimizing models and inference!) :D

Comments URL: https://news.ycombinator.com/item?id=47989158

Points: 3

# Comments: 2

Comments URL: https://news.ycombinator.com/item?id=46011367

Points: 7

# Comments: 2

As a high school student, I love LLMs and the execution of Apple's Appel Intelligence Writing Tools, but was disappointed that nothing like that existed for Windows. I thus created Writing Tools, a better than Apple Intelligence open-source alternative that works system-wide on Windows. It's much better than the tiny 2B parameter Apple Intelligence model!

Features (works system-wide): 1. Select text in a textbox, press the hotkey, and optimise your text in a click (proofread, rewrite, tone changes, custom instructions...). You can also select code to fix/refactor it with the custom instructions. 2. Press the hotkey without selecting any text to bring up a chat UI with the LLM. 3. Select all text on a webpage, press the hotkey, and use the Summarise button to get summary of the webpage/text!

It can use the free Gemini API, or a multitude of local LLMs via Ollama, llama.cpp, KoboldCPP, TabbyAPI, vLLM, etc.

It's completely free & open source, and is heavily privacy-focused (local LLM support, your API key stays on-device, no logging, no tracking, etc.)

GitHub (with a demo video): https://github.com/theJayTea/WritingTools

Writing Tools has also been featured on XDA, Beebom, Windows Central, Neowin, and more! :D

P.S. This is my first major Windows coding project, so I'm especially keen on advice for best practices and potential improvements! Thank you so much for your time :)

Comments URL: https://news.ycombinator.com/item?id=42193478

Points: 6

# Comments: 0

As a high school student, I love LLMs and the execution of Apple's Appel Intelligence Writing Tools, but was disappointed that nothing like that existed for Windows. So, I created Writing Tools, a better than Apple Intelligence open-source alternative that works system-wide on Windows! It can use the free Gemini API, or a multitude of local LLMs via Ollama, llama.cpp, KoboldCPP, TabbyAPI, vLLM, etc. It's much better than the tiny 2B parameter Apple Intelligence model! It works in any application with a customizable hotkey - Proofreads, rewrites, summarizes, and more - Free and privacy-focused (your API key stays local, no logging, no tracking, local model options, etc.)

It's built with Python and PySide6, and I've made it easy to install with a pre-compiled exe. For the technically inclined, the well-documented source is available to run or modify.

I'd love feedback from the HN community on both the concept and the implementation. Are there features you'd like to see? Any thoughts on making it more robust or user-friendly?

GitHub: https://github.com/theJayTea/WritingTools

P.S. This is my first major Windows coding project, so I'm especially keen on advice for best practices and potential improvements! Thanks so much for your time :)

Comments URL: https://news.ycombinator.com/item?id=41896518

Points: 2

# Comments: 0

Key features: - Uses Google's Gemini 1.5 Flash model for superior AI-assisted writing (much better than the tiny 2B parameter Apple Intelligence model) - Works in any application with a customizable hotkey - Proofreads, rewrites, summarizes, and more - Free and privacy-focused (your API key stays local, no logging, no tracking, etc.)

It's built with Python and PySide6, and I've made it easy to install with a pre-compiled exe. For the technically inclined, the well-documented source is available to run or modify.

I'd love feedback from the HN community on both the concept and the implementation. Are there features you'd like to see? Any thoughts on making it more robust or user-friendly?

GitHub: https://github.com/theJayTea/WritingTools

P.S. This is my first major Windows coding project, so I'm especially keen on advice for best practices and potential improvements! Thanks so much for your time :)

Comments URL: https://news.ycombinator.com/item?id=41864096

Points: 9

# Comments: 1