Since my team is mostly remote running LLM on a cluster in the office is not really viable short term.

I was considering having something run locally within out building but the time when something like that would be avaliable is not near term so i am trying to make the best of what i can do.

Reasoning is cost apparently our monthly Claude bill has become astronomical for the org. Nearly 3x our saas's cloud spend.

Apparently we are going to get limited access to codex at severely reduced plans.

I have tried some local models such as Kimi, however most are barely functional.

I am very concerned as the expectation of amount of work done is to remain consistent. Ignoring the fact teams have made entire workflows around Claude I am very worried and frustrated.

How can I help my team ease this transition? Are their local models that run well on local machines that only have 16gb ram?

Comments URL: https://news.ycombinator.com/item?id=48189073

Points: 10

# Comments: 13

When I am working with Claude I am often doing it from the root directory of a workspace of dozens of repos. I work with Claude to come up with a plan for implementing a feature and it investigates and plans.That plan often encompasses multiple repositories. Claude then turns large scale plans into smaller issues, or tickets as artifacts.

However I can honestly say anthropic is pretty terrible about support, to even billing. My org has a large enterprise contract with anthropic and we have been hitting endless rate limits across the entire org. They have never once responded to our issues, or we get the same generic AI response.

So odds of them addressing issues or responding to people feels low.

I can summarize your entire essay as frankly:

"We can give maintainers of OSS projects money to maintain projects" revolutionary never been done before. /S

I had a sales rep even call me up basically trying to book a 3 hour session to review the AI findings unprompted. When I looked at the nearly 250 page report, and saw a critical IIS bug for Windows server (doesn't exist) existing at a scanned IP address of 5xx.x.x.x (yes an impossible IP) publically available in AWS (we exclusively use gcp) I said some very choice words.

Often time tickets get fleshed out or requirements change. He just throws everything out and reshoves it into Claude.

I weep for the planet.

He is also one of our worst performers.

On large codebases I often find it taking 20+ minutes to do basic things like writing tests.

Way too often people are like it takes 2 minutes for it to do a full pr. Yeah how big is the code base actually.

I also have a coworker who is about 10x more then everyone else. Burning through credits yet he is one of the lowest performers.{closing in on around 1k worth of credits a day now).

We dropped milvus after they started trying for force their zilliz garbage saas down our throats.

One of the biggest issues is handling renewals at scale, and I hate it. Another increasingly frusturation is challenges via DNS are not quick.

I remember when my daughter wanted to play Roblox with some friends I sure as shit did my best to monitor and lock down that horrible thing. Same with just general internet monitoring. Whenever she wants to play some game or something I research it.

I have sat down with her countless times and yeah she has broken my trust a few times and she looses access to the internet.

This was not a low end place back in 2023. By the end of the last year not only had the cost increased by 30%. Internally there was a number of layoffs and quality shot down. Originally it was supposed to be 1 registered nurse per 4 people. By the end it was 1 "medical professional" per 20 who constantly rotated. Who would not answer if they had formal medical training.

Before at night people could go for walks on the premises, or if they were hungry they could go get hot food prepared by staff. New management locked everyone in at 8pm. If you missed any meal time you were told cope. Food was not allowed in the "suites" anymore. My dad and his friends there would go get together and spend time at night or socialize. Not allowed anymore after 8pm you were locked in your room.

I pulled my father out of there when he told me the above. Now his new retirement home is now also changing ownership.

They ended up switching to a terrible model with a previous release where if you were a business or in anyway making money you now needed to pay for licenses and it was comically expensive.

The reality is I could have forked it but I don't have the time and patience to deal with everything that comes from a massive project.