Over the past year we've heard the same thing from our users and community, they want the fastest and litest AI gateway.

This change allows us to address two of the most common problems we hear from users latency spikes under load and memory leaks/OOM kills that take pods down

We believe a Rust hot path is faster and bounded in memory, so those whole classes of issues go away.

It will be a gradual, non-breaking change. The Python SDK and proxy stay exactly the same, under the hood they start calling the Rust binary through PyO3, one component at a time, each proven in production before the next. The sub-1ms figure is gateway overhead (what we add on top of the upstream call), and we're aiming for a sub-100MB binary. Happy to share benchmark methodology if folks want to poke at it.

The whole gateway will be running on Rust by December 1, 2026.

Full announcement: https://docs.litellm.ai/blog/litellm-rust-launch

lite-harness has one function - query()

import { query } from "@lite-harness/sdk";

for await (const message of query({ prompt: "Fix the failing test", options: { // swap harness between: "claude-agent", "openai-agents", "pi-ai" harness: "openai-agents", model: "gpt-5.5", }, })) { console.log(message); }

Comments URL: https://news.ycombinator.com/item?id=48379288

Points: 2

# Comments: 2

Comments URL: https://news.ycombinator.com/item?id=48155595

Points: 3

# Comments: 0

The compromised PyPI packages were litellm==1.82.7 and litellm==1.82.8. Those packages have now been removed from PyPI. We have confirmed that the compromise originated from the Trivy dependency used in our CI/CD security scanning workflow. All maintainer accounts have been rotated. The new maintainer accounts are @krrish-berri-2 and @ishaan-berri. Customers running the official LiteLLM Proxy Docker image were not impacted. That deployment path pins dependencies in requirements.txt and does not rely on the compromised PyPI packages. We are pausing new LiteLLM releases until we complete a broader supply-chain review and confirm the release path is safe.

From a customer exposure standpoint, the key distinction is deployment path. Customers running the standard LiteLLM Proxy Docker deployment path were not impacted by the compromised PyPI packages.

The primary risk is to any environment that installed the LiteLLM Python package directly from PyPI during the affected window, particularly versions 1.82.7 or 1.82.8. Any customer with an internal workflow that performs a direct or unpinned pip install litellm should review that path immediately.

We are actively investigating full scope and blast radius. Our immediate next steps include:

reviewing all BerriAI repositories for impact, scanning CircleCI builds to understand blast radius and mitigate it, hardening release and publishing controls, including maintainership and credential governance, and strengthening our incident communication process for enterprise customers.

We have also engaged Google’s Mandiant security team and are actively working with them on the investigation and remediation.

OpenSwarm uses LiteLLM to add support for any LLM AnthropicAI, MistralAI, Ollama, Huggingface, GroqInc, Replicate

Comments URL: https://news.ycombinator.com/item?id=41821373

Points: 2

# Comments: 1

It's really useful to be able to specify the search space for a specific query (example: Canary allows search for the query "sagemaker" on our docs or on our github issues )

- LiteLLM Proxy IS Exactly Compatible with the OpenAI SDK

Comments URL: https://news.ycombinator.com/item?id=37826322

Points: 3

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=37578779

Points: 1

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=37447678

Points: 4

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=37438632

Points: 6

# Comments: 1

Stop relying on benchmarks and easily test LLMs in production. Try it here: https://admin.litellm.ai/

LiteLLM allows you to simplify calling any LLM as a drop in replacement for gpt-3.5-turbo

We're launching `completion_with_split_tests` to easily A/B test all LLMs.

Example usage - 1 function: completion_with_split_tests(

  models={
 "claude-2": 0.4, 
 "gpt-3.5-turbo": 0.6
  }, 

  messages=messages,

  temperature=temperature

For each completion call we allow you to:

- Control/Modify LLM configs (prompt, temperature, max_tokens etc without needing to edit code)

- Easily swap in/out 100+ LLMs without redeploying code

- View Input/Outputs for each LLM on our UI

- Retry requests with an alternate LLM

Happy completion()!

Comments URL: https://news.ycombinator.com/item?id=37362967

Points: 2

# Comments: 0

Code - https://github.com/BerriAI/litellm/blob/main/cookbook/Evalua...

Are others seeing similar results?

Comments URL: https://news.ycombinator.com/item?id=37153342

Points: 1

# Comments: 2

Should we add support for llama.cpp and vllm.ai in the proxy server ? Or should we assume you can host them on your own infra and the proxy server requests your hosted model ?

I’m the maintainer of liteLLM() - package to simplify input/output to OpenAI, Azure, Cohere, Anthropic, Hugging face API Endpoints: https://github.com/BerriAI/litellm/

We’re open sourcing our implementation of liteLLM proxy: https://github.com/BerriAI/litellm/blob/main/cookbook/proxy-...

TLDR: It has one API endpoint /chat/completions and standardizes input/output for 50+ LLM models + handles logging, error tracking, caching, streaming

What can liteLLM proxy do? - It’s a central place to manage all LLM provider integrations

- Consistent Input/Output Format - Call all models using the OpenAI format: completion(model, messages) - Text responses will always be available at ['choices'][0]['message']['content']

- Error Handling Using Model Fallbacks (if GPT-4 fails, try llama2)

- Logging - Log Requests, Responses and Errors to Supabase, Posthog, Mixpanel, Sentry, Helicone

- Token Usage & Spend - Track Input + Completion tokens used + Spend/model

- Caching - Implementation of Semantic Caching

- Streaming & Async Support - Return generators to stream text responses

You can deploy liteLLM to your own infrastructure using Railway, GCP, AWS, Azure

Happy completion() !

Comments URL: https://news.ycombinator.com/item?id=37095542

Points: 140

# Comments: 34

Comments URL: https://news.ycombinator.com/item?id=37048149

Points: 5

# Comments: 1