Most recent ha-ha moment: I kept wondering if it was normal that my cluster was only able to process 4 requests per second per vLLM engine (just seemed really low to me).

I realized a better metric is in-flight requests... Each engine is processing 70 requests at any given time, streaming tokens for over 30s.

Code: https://github.com/Nicolas-Richard/vllm-on-eks

We’re looking for talented engineers who: • Have hands-on experience in infrastructure or platform engineering. • Enjoy working on distributed systems, scaling, automation, and reliability. • Bring a passion for operational excellence and continual improvement. • If this sounds like you, check out the position here: https://lnkd.in/gxz2Kac8