EEBench is a set of original EE tasks: design-from-spec, datasheet comprehension, design review, picking real parts that actually exist. Agents submit design source, not prose. The grader runs compiler checks and simulations, and scores measured behavior — transients, thresholds, tolerance corners — against the spec. No human graders, no LLM-as-judge. When a score says the rail sagged below 3.0 V during brown-out, that's a waveform, not an opinion.

Things that surprised us:

The frontier cluster spans roughly 45–72% (Claude Fable 5 on top at 71.7%). Open-weight models that do well on coding benchmarks score 5–13% here. Cost per task varies 15x between models with similar scores. Honest limitations: the task set is curated from a larger pack to separate the frontier models, and it's growing. The pack is held out to keep it out of training data, so you can't self-run it yet. If you want a model on the board, email us and we'll run it.

Happy to go deep on the grading harness — ask us anything.

Comments URL: https://news.ycombinator.com/item?id=48766320

Points: 1

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=47828246

Points: 2

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=44639802

Points: 24

# Comments: 3

Currently working on a pipeline to generate a whole bunch of these automatically, stick them on some big test boards and make sure they actually work. We will be selling razor blades, and will have the test data to show they work.

We already provide the 'plug and play' version which looks quite alot like LCSC data and is certainly good enough for playing around with. Id really like to put some effort into standardizing this in the mid-term, seems pretty crazy to me that there are way more footprints designs out there than actual packages.

The schematic is not most of the work, but it is the place that most of the work is relying on as a source of truth. Everything from firmware header files, end of line testing, reliability tests to service manuals. Today as an engineer you end up in the middle of that. Once you have a complete description, much of the boiler plate work can be automated.

We have done quite alot on the components and routing section of the work as well, we can generate most components for you automatically without leaving the IDE and have a growing list of fully auto-selected components. Instead of going to digkey to find a 10kohm 1% 0402, you can just give those specs to our tool and it will automatically pick the most suitable one for you, considering things like price and stock (plus eventually any other filtering you might care about, including custom libraries). For layout we currently just to layout reuse blocks, which is actually a crazy time saver for how simple it is. https://packages.atopile.io/ currently has ~10s of parts, but we are growing it over the next few weeks to ~thousands, so most parts you want to use will have a reference design + layout you can install and use in your project, just like you would NPM/pypi.

For example the INA228 from TI has a funky addressing scheme that is not possible to describe in ato, so we create a hybrid module:

atopile for the main structure: https://github.com/atopile/packages/blob/multi/adafruit_heis...

compose in an 'addressor' python module: https://github.com/atopile/packages/blob/multi/adafruit_heis...

Here is one of our more recent demo projects you can check out that shows how a larger project is structured: https://github.com/atopile/nonos/blob/main/elec/src/nonos.at...

And an example of modeling a component: https://github.com/atopile/packages/tree/main/packages/archi...

Not personally a fan of the readability of lisp, but that is just a personal preference. Our assumption is that python is going to be the most familiar language to our users and following a python like syntax will make picking it up easy. Our language is definately still quite immature and we are still figuring out exactly what it should look like.

There is definitely a little bit of a learning curve at the moment, we do have some getting started videos that walk you through the whole process. If there is anything missing or confusing, please point it out! We do want to make it as approachable as possible. It will only get better from here, I promise!

On the downloading, currently we have the package on pypi, which does make it pretty easy to install from a command line, but I can appreciate that will be new to alot of people. In the future we might do an executable download version.

We have a getting started video here: https://www.youtube.com/watch?v=7aeZLlA_VYA

We are very active on discord also if you get stuck or have any questions/feedback, hope to see you there! https://discord.gg/PBq4pS4K3p

We currently support components on JLC, but eventually plan to build out a pretty substantial library. We will also capture a good fraction of information that you would currently need to go to the datasheet for.

BOM generation and abstraction have definitely not made it to PCBA design yet, so that is where we are starting.

Stoked to get to the hard stuff.

Being able to use git was our fundamental motivator, we all previously worked a big companies and found it maddening that we couldnt work in parallel on projects without breaking everything.

On parts specs, for sure, we currently capture all the data you would see on JLCs website. We do have dielectric and voltage ratings for caps. Eventually we plan to scrape a bunch of datasheets to build out a high quality dataset, I am very excited about this!

Units and tolerances are core to our language, the physical world is 'fuzzy' and having a good way to deal with those we think is pretty important.

We are also trying to make it as readable and friendly as possible, our expectation is our users will likely have some experience with python and perhaps a little C back in school, so making it clear and approachable is front of mind.

Very open to critiques on our choices! We still very much in development.