I modeled this as a fleet scheduling problem using Queueing Theory and Dynamic Programming over a finite horizon. The standard practice of throttling once the number of jobs in server exceeds certain threshold is in fact suboptimal. The optimal policy consists in segmenting the part of the traffic that is retry sensitive, from those that are not. For example, an user doing a basic data parsing might still do well under a weaker model, but a power user will certainly feel the degradation and ask more.

Demo: Just a toy instance to illustrate the issue. The user can create their own policies and see how they perform against the industry standard and the optimal one. It is roughly 100 lines of Flask + JS frontend.

Paper with proofs: https://arxiv.org/abs/2608.23986

For those of you who have worked in inference infra, does this match anything you have seen?

Comments URL: https://news.ycombinator.com/item?id=49521092

Points: 2

# Comments: 0

The idea seems natural: as the data center demand increases momentarily through the day, throttling their models (either using quantized versions, reducing the context window or lowering the tier of the model to a smaller one) seems appealing as the replacement model in principle uses less electricity. The problem is that this can cause the opposite effect: as users are trying to solve a question, if the degraded AI model gives a bad answer, the user is likely to keep re-asking. On the AI provider side this looks paradoxical: throttling to a lower model creates in fact more demand for their data center.

This problem is even worse for agentic workflows, as these are more likely to create a re-ask storm, and maybe explains the outages and anecdotal experiences of users that feel the models are degraded.

The model: I used mainly queueing theory arguments solving the optimal scheduling serving for an AI fleet with heterogeneous users solving a finite horizon Dynamic Programming optimization problem.

Insights: The industry standard practice of throttling once the number of users in system exceeds a given threshold is in fact what’s causing the problem, the optimal rule implies separating users that won’t feel degradation as much with users that are very sensitive to it (agents and power users vs users doing simple tasks).

Limitations: The visualization and paper examples are a toy example to illustrate the problem, only the providers have enough data to properly calibrate these instances. In the paper there are some interesting calibrated instances.

Technical Details: The visualization is around 100 lines of flask plus js frontend (LLM assisted with ground truth based on the original numerical example of the paper).

Paper with proofs/theory: https://arxiv.org/abs/2608.23986

Comments URL: https://news.ycombinator.com/item?id=49477620

Points: 6

# Comments: 0

The idea seems natural: as the data center demand increases momentarily through the day, throttling their models (either using a quantized version, reducing the context window or lowering the tier of the model to a smaller one) seems appealing as the model in principle uses less electricity. The problem is that this can cause in fact the opposite effect: as users are trying to solve a question, if the AI gives a bad answer, the user is likely to keep re-asking. On the AI provider side this looks paradoxical: throttling to a lower model creates in fact more demand for their data center.

This problem is even worse for agentic workflows, as these are more likely to create a re-ask storm, and maybe explains the outages and anecdotal experiences of users that feel the models are degraded.

The model: I used mainly queueing theory arguments solving the optimal scheduling serving for an AI fleet with heterogeneous users.

  Insights: The industry standard practice of throttling once the number of users in system is in fact what’s causing the problem, the optimal rule implies separating users that won’t feel degradation as much with users that are very sensitive to it.  

  Technical Details: The visualization is around 100 lines of flask plus js frontend (LLM assisted with ground truth based on the original numerical example of the paper).  

Comments URL: https://news.ycombinator.com/item?id=49448480

Points: 3

# Comments: 0

Think of the 2008 crisis where companies that had nothing to do with Credit Default Swaps were exposed to the crash due to network effects, potentially crashing the whole economy.

Then, the government intervened by bailing the banks out. Here, it's not really clear if there is even an effective intervention.