Openlayer is solving the AI reliability problem. We are looking for stellar customer engineers to join the team and help make customers happy. Think air-traffic control for our customers: you will be in charge of building requested features, integrations, and fixing bugs.

We are hiring in SF, NYC, and open to remote. Our team is currently 6 people, but we are very actively growing and adding more headcount. Those with experience in Python, Next.js, Docker, and cloud compute systems are encouraged to apply!

https://www.ycombinator.com/companies/openlayer/jobs/yIE9WI3...

Openlayer is solving the AI reliability problem. We are looking for stellar frontend engineers to join the team and help build the platform that next-gen teams use to build the most powerful AI systems.

We are hiring in SF, NYC, and open to remote. Our team is currently 6 people, but we are very actively growing and adding more headcount. Those with experience in Next.js, React, and TypeScript are encouraged to apply!

https://www.ycombinator.com/companies/openlayer/jobs/ZEEO8UB...

Most of us get how crucial AI evals are now. The thing is, almost all the eval platforms we've seen are clunky and don't have any product cohesion. There's too much manual setup and adaptation needed, which breaks developers' workflows.

That's why we're releasing a simpler workflow.

If you're using GitHub, you only need to add two files to the repo (one config + one script). Then, connect your repo to Openlayer and define must-pass tests for your AI system. Once integrated, every commit triggers these tests automatically on Openlayer, ensuring continuous evaluation without extra effort.

We offer 100+ tests (and are always adding more), including custom tests. We're language-agnostic, and you can customize the workflow using our CLI and REST API.

As a final note, you can leverage the same setup to monitor your live AI systems after you deploy them. It's just a matter of setting some env vars in your staging/prod environments, and your Openlayer tests will run on top of your live data and send alerts if they start failing.

Let us know what you think!

Comments URL: https://news.ycombinator.com/item?id=40158283

Points: 8

# Comments: 1

We offer more features around error and subpopulation analysis, versioning, running evals during development, and collaboration. Through what (I believe) is a more clean and simple DevEx and UI!

re: Lilac, there’s some intersect w/r/t dataset exploration, but we have more evals than the ones they offer. More than data quality, we give insights into data drift and model performance and let you set up expectations and get alerts on whether they fail during development and production. + distinct in some of the ways described above

We’re really happy to see more tools and platforms in this space. Definitely a big uptick since we started 3 years ago, w the advent of gen ai this is all top of mind (and deservedly so).

If you want to unlock explainability for your tabular classification or regression, or text classification models, you can upload the actual model binary. We support a bunch of frameworks out-of-the-box, but you can use any architecture through our custom upload.

More info:

https://docs.openlayer.com/documentation/how-to-guides/uploa...

https://docs.openlayer.com/documentation/how-to-guides/write...

We noticed that the industry is laser-focused on tackling feature drift after production, but we spotted a gap. Most ML teams are wrestling with model validation even before anything is deployed in production. We also noticed that post-deployment analysis sometimes misses the mark, lacking components like identifying underperforming cohorts or giving actionable insights. This leads to a barrage of alerts and the inevitable alert fatigue.

We decided to start Openlayer to offer a more holistic solution that helps teams from the ML development and experiment tracking phase to more advanced tasks like monitoring and fairness. We established a strong baseline with this launch and are now building several features on top.

Stay tuned!

Some common ones are fraud and churn detection for financial institutions or e-commerce sites (both tabular classification examples). It's very important for these types of tasks in particular to guard against biases and false negatives, so they use us to set up wide test nets that help give them assurance that their models are working properly before they hit production (and to monitor them post-deployment).

Another example is Zuma (https://www.getzuma.com), a startup building an AI-driven chatbot that uses us to track their experiments and improve the accuracy of their NLP intent classification model.

Of course, we're also building out support for evaluating LLMs. Because this is an open problem, we've been spending a lot of time interviewing people in the space who are building these models (please reach out if this is you!).

Openlayer is an ML testing, evaluation, and observability platform designed to help teams pinpoint and resolve issues in their models.

We were ML engineers experiencing the struggle that goes into properly evaluating models, making them robust to the myriad of unexpected edge cases they encounter in production, and understanding the reasons behind their mistakes. It was like playing an endless game of whack-a-mole with Jupyter notebooks and CSV files — fix one issue and another pops up. This shouldn’t be the case. Error analysis is vital to establishing guardrails for AI and ensuring fairness across model predictions.

Traditional software testing platforms are designed for deterministic systems, where a given input produces an expected output. Since ML models are probabilistic, testing them reliably has been a challenge. What sets Openlayer apart from other companies in the space is our end-to-end approach to tackling both pre- and post-deployment stages of the ML pipeline. This "shift-left" approach emphasizes the importance of thorough validation before you ship, rather than relying solely on monitoring after you deploy. Having a strong evaluation process pre-ship means fewer bugs for your users, shorter and more efficient dev-cycles, and lower chances of getting into a PR disaster or having to recall a model.

Openlayer provides ML teams and individuals with a suite of powerful tools to understand models and data beyond your typical metrics. The platform offers insights about the quality of your training and validation sets, the performance of your model across subpopulations of your data, and much more. Each of these insights can be turned into a “goal.” As you commit new versions of your models and data, you can see how your model progresses towards these goals, as you guard against regressions you may have otherwise not picked up on and continually raise the bar.

Here's a quick rundown of the Openlayer workflow:

1. Add a hook in your training / data ingestion pipeline to upload your data and model predictions to Openlayer via our API

2. Explore insights about your models and data and create goals around them [1]

3. Diagnose issues with the help of our platform, using powerful tools like explainability (e.g. SHAP values) to get actionable recommendations on how to improve

4. Track the progress over time towards your goals with our UI and API and create new ones to keep improving

We've got a free sandbox for you to try out the platform today! You can sign up here: https://app.openlayer.com/. We are also soon adding support for even more ML tasks, so please reach out if your use case is not supported and we can add you to a waitlist.

Give Openlayer a spin and join us in revolutionizing ML development for greater efficiency and success. Let us know what you think, or if you have any questions about Openlayer or model evaluation in general.

[1] A quick run-down of the categories of goals you can track:

- Integrity goals measure the quality of your validation and training sets

- Consistency goals guard against drift between your datasets

- Performance goals evaluate your model's performance across subpopulations of the data

- Robustness goals stress-test your model using synthetic data to uncover edge cases

- Fairness goals help you understand biases in your model on sensitive populations

Comments URL: https://news.ycombinator.com/item?id=35951703

Points: 53

# Comments: 11

My name is Vikas, and my cofounders Rish, Gabe and I are building [Openlayer](http://openlayer.com/?ref=hn). Openlayer is an ML testing, evaluation, and observability platform designed to help teams pinpoint and resolve issues in their models.

We were ML engineers experiencing the struggle that goes into properly evaluating models, making them robust to the myriad of unexpected edge cases they will encounter in production, and understanding the reasons behind their mistakes. It was like playing an endless game of whack-a-mole with Jupyter notebooks and CSV files — fix one issue and another pops up. This shouldn’t be the case. Error analysis is vital to establishing guardrails for AI and ensuring fairness across model predictions.

Traditional software testing platforms are designed for deterministic systems, where a given input produces an expected output. Since ML models are probabilistic, testing them reliably has been a challenge. What sets Openlayer apart from other companies in the space is our end-to-end approach to tackling both pre- and post-deployment stages of the ML pipeline. This "shift-left" approach emphasizes the importance of thorough validation in pre-production rather than solely focusing on monitoring after deployment. Having a strong evaluation process pre-ship means fewer bugs in production, shorter and more efficient dev-cycles, and lower chances of getting into a PR disaster or having to recall a model.

Openlayer provides ML teams and individuals with a suite of powerful tools to understand your models and data beyond simple metrics. The platform offers insights about the quality of your training and validation sets, the performance of your model across subpopulations of that data, and much more. Each of these insights can be turned into a “goal.” As you commit new versions of your models and data, you can see how your model progresses towards these goals, as you guard against regressions you may have otherwise not picked up on and continue to raise the bar.

Here's a quick rundown of Openlayer's workflow:

1. Add a hook in your training / data ingestion pipeline to upload your data and model predictions to Openlayer via our API

2. Explore insights about your models and data and create goals around them - *Integrity* goals track the quality of your validation and training sets - *Consistency* goals guard against drift between your datasets - *Performance* goals evaluate your model's performance across subpopulations of the data - *Robustness* goals stress-test your model using synthetic data to uncover edge cases - *Fairness* goals help you understand biases in your model on sensitive populations

3. Diagnose issues with the help of our platform, using powerful tools like explainability (e.g. SHAP values) to get actionable recommendations on how to improve

4. Track the progress over time towards your goals with our UI and API and create new ones to keep improving

Openlayer works best for teams — technical and non-technical folk can all play a role in the process of understanding, goal-setting, and improving. The platform enables this through a user-friendly UI (and API) with comments and notifications built-in to help keep everyone on the same page.

We're excited to offer a free sandbox for you to try out the platform today! [You can sign up here](https://app.openlayer.com/?ref=hn). We are also soon adding support for even more ML tasks, so please reach out if your use case is not supported and we can add you to a waitlist.

Give Openlayer a spin and join us in revolutionizing ML development for greater efficiency and success. Let us know what you think, or if you have any questions about Openlayer or model evaluation in general.

Comments URL: https://news.ycombinator.com/item?id=35886396

Points: 14

# Comments: 2

My mother is a physician, she just last night told me about a case she saw over the weekend in which a young 20-something nearly OD’d on fentanyl from taking ecstasy. She survived, but with life-altering brain trauma rendering her unable to remember who she is. She needs tubes for her food supply, and a ventilator to breathe.

Here you can see a few images of what is a lethal dose of fentanyl: https://www.dea.gov/galleries/drug-images/fentanyl

Scary stuff, and maybe there’s an argument there to be made in favor of legalizing current black-market drugs. Definitely a PSA to test your drugs.

CDC on cross-contamination: https://www.cdc.gov/drugoverdose/deaths/other-drugs.html

How simple is it to install Payload in an existing Next.js app to power basically just the /blog/* subdirectory?

Have you thought about making this work for more things than ArXiv? Could be interesting to prepend Smort to any URL and create “rooms” for annotating the web (other static content like Wikipedia, or even temporal content like Youtube videos). But maybe that’s too wide of a scope, just got me thinking :)

Tons of potholes like this exist in AI we use everyday. We used to be ML engineers at Siri and had to invest millions into monitoring tools to stay on top. This is fine and all, but what’s better is to catch them before you ship and before your users suffer (sometimes literally, as in this case).

We think that better tools for QA-ing models, which allow more people (not just ML engineers) to get eyes on the model, might help catch mistakes proactively rather than retroactively.

https://haseebq.com/about/

Also see this comment below, something I wasn’t aware of: https://news.ycombinator.com/reply?id=20212953&goto=item%3Fi...

I’m curious- at the end of the article, one of the scientists remarks that the stars might be “zipping around” their host star faster than measurements predict, which could rule out potential for life. Is this referring to your suspicion re: proximity => tidal locking => extreme temperatures? Or is there some way that speed of orbit can affect potential for life?

The list of possible extenuating circumstances extends to infinity when you think about it more deliberately. If you don’t, you’ve successfully bucketed someone based on quick, irrational judgment. That is poor behavior.

Location: USA

Remote: No

Willing to relocate: Yea

Technologies: Python/Django, Swift, Java, Javascript/Node/ Express/React

Resume/CV: https://www.linkedin.com/in/vikasnair/

Email: me@vikasnair.com