Context is we didn't think these would work, and did look quite closely. We may have missed something though.

Not to say this post isn't AI generated but you might want a better tool (if one exists)

In a past life I’d struggled a lot with a public API that had some really tricky pagination performance problems. It was something we were always fighting, be it awkward edge case data shapes or the Postgres planner having bad statistics, where everything would get difficult past >5TB in the table.

Was really happy to see the team find a solution that feels scalable and can be generically applied to a lot of our endpoints at incident. Quite a great outcome where I think we’ll be safely scalable for the next few years instead of hitting other problems that would crop up had we gone with gin indexes or otherwise.

Happy to have fixed up a bunch of these issues already though.

We've since built on-call directly into our product and we've had loads of those customers migrate entirely into us, dropping PagerDuty. The biggest customers we're onboarding now tend to buy us as their sole on-call and incident management tool, too.

We have a bunch of case studies of people who've moved if that's useful to anyone. My favourite is probably Intercom who migrated from PagerDuty into our on-call in a few weeks (the Intercom team are great fun to work with!)

https://incident.io/customers/intercom

I thought it would be useful to share, as we have evaluation metrics and scorecards for investigations, so you can see how real-world performance compares between models.

I've written the post on LinkedIn so I could share a picture of the scorecards and how they compare:

https://www.linkedin.com/posts/lawrence2jones_like-many-others-we-were-excited-about-openai-activity-7317907307634323457-FdL7

Our takeaways were:

- 4.1 is much fussier than Sonnet 3.7 at claiming a code change caused an incident, leading to a drop (38%) in recall

- When 4.1 does suggest a PR caused an incident, it's right 33% more than Sonnet 3.7

- 4.1 blows 4o out the water, with 4o finding just 3/31 of the code changes in our dataset, showing how much of an upgrade 4.1 is on this task

In short, 4.1 is a totally different beast to 4o when it comes to software tasks, and at a much lower price-point than Sonnet 3.7 we'll be considering it carefully across our agents.

We are also yet to find a metric where 4.1 is worse than 4o, so at minimum this release means >20% cost savings for us.

Hopefully useful to people!

Comments URL: https://news.ycombinator.com/item?id=43698187

Points: 1

# Comments: 0

I work at incident.io it's a daily struggle.

Would be interested if anyone else sees these dynamics playing out. It seems there’s a really powerful combination of larger players starting later and a rising tide of AI improvements that mean the smaller shops can leap frog ahead, in ways that I haven’t seen before.

There’s a first wave of incident startups that responded to the market having stagnated about 4 years ago (incident.io, FireHydrant, Rootly) then a slew of extremely recent (<1 year) companies leaning into AI incident response.

It’s weird that Opsgenie is just quitting that race but realistically they weren’t really competing in terms of pace of development. Felt more like Opsgenie was bought under the assumption IR was a ‘solved’ problem that Atlassian could just add to their stack and be done with it, while today it’s increasingly apparently that just paging someone is the smallest part.

We’ve got a good migration story (import OG schedules and escalation paths, etc) so most customers just migrate to us, but if you didn’t have that option the Atlassian migration is much more painful.

The writing has been on the wall for a while with Opsgenie though. I work at incident.io and loads of our customers have had deprecation/migration warnings for a year or so now, pushing them away from Opsgenie and into different Atlassian products (with no migration plan, sadly).

It’s been great for us as they tend to move to us (we have good migration support for Opsgenie) but confusing to watch from the outside as Atlassian never stopped selling the product even while actively shutting accounts down.

Pretty reductive argument honestly. AI services mean that a lot of AI operations are 'just HTTP requests' but it doesn't mean there isn't a huge amount of tooling that you need to work effectively with them.

AI tooling in this case being:

- Fine-tuning - Eval test suites - Observability based around AI interactions

Would you call Lovable a basic app that makes API calls? Or is it AI enough to justify the label?

Asking as they made a move from Python to Go just the other day for similar reasons as in this post: https://lovable.dev/blog/from-python-to-go

I wouldn't agree exactly with the article, in that while it's very easy to start making decisions that are genuinely ROI silly (all companies make them, my current one not excluded) there is a balance between just paying without question and adding friction that encourages good decision making.

But in our case, getting the data wasn't too much effort, and helped inform us for a subsequent batch of hardware purchases. It ended up representing about $50k of spend that we'd like to allocate well, and took me a couple of days to investigate and write-up: my day rate means that was well worth it.

But for SaaS in general I think the trend is noticeable? Twitter led the way with massive layoffs in engineering often in the roles around reliability. The industry as a whole have aimed to cut costs however possible, and reliability/ops is usually seen as a cost-centre that gets hit hard.

I've watched this in my own space (start/scale-ups and larger companies, I work in incident response tooling) as people start talking very differently about reliability and engineering investment. You hear "do more with less" about five times every day and spend that was previously greenlit by default around reliability/redundancy is under much more scrutiny now.

I see this as a silent mirror of the reduction in open-source efforts from companies now there's been a refocus on business impact and bottom-line.

GitHub laid off 10% of their staff in 2023 and like you say, won't have slowed down to account for that.

Ignoring all that though... other than Copilot, what big feature changes have you seen in GitHub? My experience of using their product has been broadly unchanged for years.

It's no surprise we're now feeling those effects but damn, GitHub and other services like Slack have been really bad lately.

If you store contexts on your structs it’s very likely you won’t thread them correctly, leading to errors like database code not properly handling transactions.

Actually super fragile and you should avoid doing this as much as is possible. It’s never a good idea!