I reached out to a lot of other inference providers such as fireworks, togetherAI, simpliAI etc and started asking them their growth and what they are seeing in this space / what they predict we will see over this year.

I was told by a higher up at fireworks that on average since January the space as a whole has had 10% week over week growth -- this type of explosive growth feels unreal. I honestly expect a price squeeze later this year, no one can keep up.

I think we're starting to see this with GPU prices -- h100s have gone from 1.30 on demand to 1.90 about, h200 nodes are all sold out with 3+ month waitlists. B300s have near a year waitlist from the quotes I got. Insane market.

Comments URL: https://news.ycombinator.com/item?id=47906463

Points: 2

# Comments: 1

Wrote a short piece on GPU waste. We've been talking with many enterprises and neoclouds for the past 9 months to get a good understanding of utilization rates (and they are shockingly low).

Most clusters sit with less than half of their GPUs utilized, translating to billions in lost compute time. Our goal is to help solve this problem at Lilac. Let me know what your thoughts are!

Comments URL: https://news.ycombinator.com/item?id=47524874

Points: 5

# Comments: 2