Hacker News: janaksunil

New comment by janaksunil in "Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases"

janaksunil — Sun, 13 Sep 2026 07:17:32 +0000

do you have some time to chat? janak@withspecific.com

New comment by janaksunil in "Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases"

janaksunil — Sun, 13 Sep 2026 06:47:34 +0000

would love to chat and learn more about your set up! here's my email - janak@withspecific.com

New comment by janaksunil in "Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases"

janaksunil — Sun, 13 Sep 2026 06:42:07 +0000

we manually vet all codebases and companies

New comment by janaksunil in "Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases"

janaksunil — Sun, 13 Sep 2026 06:41:48 +0000

yes!

New comment by janaksunil in "Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases"

janaksunil — Sun, 13 Sep 2026 06:41:19 +0000

that's fair - for long horizon engineering tasks would speed still matter?

New comment by janaksunil in "Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases"

janaksunil — Sun, 13 Sep 2026 06:40:44 +0000

this is a good question. what would make you reject an otherwise working PR on design grounds?

New comment by janaksunil in "Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases"

janaksunil — Sun, 13 Sep 2026 06:38:28 +0000

we reached out to companies that were willing to license their codebases. every codebase we used had real users, one of them had 200k+ users and is currently top 100 on the app store.

New comment by janaksunil in "Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases"

janaksunil — Sun, 13 Sep 2026 06:35:26 +0000

i appreciate the feedback, the benchmark is primarily long horizon real world engineering tasks on big private codebases.

New comment by janaksunil in "Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases"

janaksunil — Sun, 13 Sep 2026 06:34:11 +0000

we're going to open source some of our tasks and model trajectories as well

New comment by janaksunil in "Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases"

janaksunil — Sun, 13 Sep 2026 06:33:06 +0000

all the codebases were written pre-2023, so pre when AI got good at coding

New comment by janaksunil in "Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases"

janaksunil — Sun, 13 Sep 2026 06:32:25 +0000

would love to learn why?

New comment by janaksunil in "Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases"

janaksunil — Sun, 13 Sep 2026 06:32:06 +0000

will do! happy to chat more on janak@withspecific.com as well

New comment by janaksunil in "Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases"

janaksunil — Sun, 13 Sep 2026 06:31:17 +0000

we've done our best to use the native provider's harness. all models were run on 'high' reasoning. this is still v1 and tons of room for improvement - really appreciate your feedback!

New comment by janaksunil in "Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases"

janaksunil — Sun, 13 Sep 2026 06:30:11 +0000

makes sense

New comment by janaksunil in "Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases"

janaksunil — Sun, 13 Sep 2026 06:29:48 +0000

the tasks on the benchmar are long horizon swe tasks - where GLM does surprisingly well

New comment by janaksunil in "Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases"

janaksunil — Sun, 13 Sep 2026 06:28:21 +0000

here's where all the models messed up!

its under this section 'Missed requirements are the most common failure' on realswe.withspecific.com

we also have the setup in the blog. the reason for lower success rates is that we gave models ambitious tasks that real engineers worked on for weeks.

New comment by janaksunil in "Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases"

janaksunil — Sun, 13 Sep 2026 06:26:35 +0000

nope, these were private codebases

New comment by janaksunil in "Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases"

janaksunil — Sun, 13 Sep 2026 06:26:00 +0000

hey this seems really interesting - what prompted you to test multiple agents on your codebase?

New comment by janaksunil in "Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases"

janaksunil — Sun, 13 Sep 2026 06:23:57 +0000

i'm janak, cofounder of Specific Labs (YC F25) and one of the authors of Real-SWE.

if i can help answer any questions please feel free to email me at janak@withspecific.com, happy to send over my phone number as well :)

Show HN: MCP server that finds dev tool credits in your workflow

janaksunil — Fri, 06 Mar 2026 23:54:47 +0000

I made an MCP server that plugs into Claude Code and tells you if there are credits or discounts available for tools in your stack.

First version was super spammy - it fired every time you mentioned any tool. Now it only triggers when you're actually evaluating something new. Also working on having it scan your package.json on init so it can flag deals on stuff you're already paying for.