Remote: Yes

Willing to relocate: Yes

Technologies: FastAPI, Python, Nextjs, Typescript, Docker, Langchain, Langfuse, PostgresQL, Redis, AWS (ec2, s3,lambda, ecs, ebs, cloudfront), VNC (x11vnc, novnc)

Resume: https://drive.google.com/file/d/19NhGlGj-R2aLO5KOizT8Kt_GOHL...

email: janardanhazarika20@gmail.com

https://www.janardan.xyz/writing/deconstructing-ai-eval-lab-workings

I got bored of UI work at my day job and wanted to build something. Ended up building a platform that streams KiCad (a PCB design tool) to the browser via VNC, tracks what the user is doing on the board in real time, and uses an LLM to evaluate their process at the end. The idea: coding assessments exist everywhere, but nothing like this for EE/hardware folks. Wanted to see if you could evaluate an engineer by just watching them work. Still rough. VNC latency and lag is real. No proctoring yet (MVP phase). But the core thing works. Blog has the full breakdown of how it's built.

Comments URL: https://news.ycombinator.com/item?id=47913720

Points: 7

# Comments: 0

A MVP is live at this https://ai-eval-lab.janardan.xyz/

Github: https://github.com/janardannn/ai-eval-lab

For now, for every session, inside the docker container, i create a virtual frame buffer using xvfb and render KiCad in it. Thereafter using x11vnc to pickup the xvfb display and serve it using VNC. Since browsers cant use raw TCP connections, i am using websockify to bridge WS <--> TCP. To render the app in browser, I am using noVNC to render an iframe inside my app. noVNC opens a websocket to websockify and the frames from the container start flowing.

Inside, KiCad itself, a have added a python plugin which auto loads when KiCad is started as a background thread. After a set interval of time, it tracks all the board objects like footprints, tracks, zones etc and serialize them as a JSON and makes a POST request to the backend server for the specific session id.

For the grading and evaluation, as of now, I am using an LLM to cross check the entire sequential board activity that was sent over the duration of the assessment and grade it against a predefined rubric/score it against per-checkpoint criteria. This is entirely based on how the admin have set the assessment grading rubrics.

I am not building it as a business rather exploring the space and the technology. It was an excuse for me to learn about VNC, streaming, docker orchestration and also somewhat of AWS through its deployment.

I'm hosting it on the free AWS tier so only one session at a time. If you queued then please give it some time.

Happy to chat about anything regarding the project!