Comments URL: https://news.ycombinator.com/item?id=50158998

Points: 1

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=49557613

Points: 1

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=39706709

Points: 13

# Comments: 0

Debuggers still work pretty much the same as gdb, only now with an integrated UI, and there's so much room for improvement. This is a great start and can be taken much further with abilities to mimic complete sessions to easily reproduce bugs and run them again and again until fixed.

We see the same at the related testing automation field (disclaimer - checksum.ai founder). Same testing methods, same testing problems, only fancier packages.

1. We hash all inner text and then backfill static strings on the server side. So every text that is specific to the user remains hashed

2. We detect special cases like passwords, SSNs, credit cards, and completely block it (even not hashed)

3. We provide full privacy controls to our customers to easily mask any sensitive elements

4. We discard the user IP and don't require any PII to be sent. So we can connect a session together, but don't really know who the user is

We don’t have pricing, not because we try to be vague, but because we haven’t fully figured out our training costs, which can vary significantly per app. We are very much in the “Do things that don’t scale” phase where we hand-pick our customers, provide white-glove treatment and prioritize learnings over price

Then, during test generation, we bias the model to explore edge cases (in a few ways), and the model is still able to complete those even with low sample.

In other words, we direct the model toward certain goals, and flows and also add chaos to the process which result in the model executing unexpected flows.

Feel free to sign up for a demo if that's a priority for your team. Even if it's just to chat and connect.

1. We use AI to analyze the user patterns and find common paths and edge cases, basically building a representation of your UX in a DB

2. We then use the DB to train another ML model that learns how to use your app the same way a user does. Given a certain page and user context, the ML can complete UX flows.

3. Finally, we learn to generate assertions, run the tests and convert the model actions in step 2 into proper Playwright or Cypress tests

1. We (and others) have tested our tools' impact on memory, CPU, network performance and found only negligible impact, even on slower/older devices

2. Also, they are used by F500 companies and have wide adoption, which indicates that other well established devs have run the same tests and decided to move forward.

We'll work on the language there to clarify.

There's nothing "specific" in the underlying model that prevents it from testing mobile. It's just a matter of focus at the current time.

But the tl;dr is 1. We learn how to use your app based on real sessions (we remove sensitive information on the client side) 2. We train a model on this data 3. We connect this model to a browser and generate Playwright or Cypress tests

The end result is code written and Playwright or Cypress. You can edit and run the tests regularly

> How is the product different from the other test generation tools

We train our models based on real user sessions. So our tests are: 1. Completely auto-generated 2. Achieve high coverage of real user flows, including detecting edge cases 3. Automatically maintained and execute with our models so they are less flakey.

> How do you check if the are testing the intended behavior

Our models are trained on many real sessions so it learns how your website (and others) should behave. In that sense, it's similar to a manual QA tester which can detect bugs. To supplement for functionality that is not obvious by the UI, we are now looking at adding LLMs to parse code, but most of the functionality can be inferred from the UI

We actually use real user sessions to train our model, so when I use the term coverage our main metric is covering as many user behaviors as possible.

We collect data in a privacy-focused way essentially anonymizing all sensitive information, as we don't need to know the user specific context. Only the main flow.

I’m Gal, co-founder at Checksum (https://checksum.ai). Checksum is a tool for automatically generating and maintaining end-to-end tests using AI.

I cut my teeth in applied ML in 2016 at a maritime tech company called TSG, based in Israel. When I was there, I worked on a cool product that used machine learning to detect suspicious vehicles. Radar data is pretty tough for humans to parse, but a great fit for AI – and it worked very well for detecting smugglers, terrorist activity, and that sort of thing.

In 2021, after a few years working in big tech (Lyft, Google), I joined a YC company, seer W21, as CTO. This is where I experienced the unique pain of trying to keep end-to-end tests in a good state. The app was quite featureful, and it was a struggle to get and maintain good test coverage.

Like the suspicious maritime vehicle problem I had previously encountered, building and maintaining E2E tests had all the markings of a problem where machines could outperform humans. Also, in the early user interviews, it became clear that this problem wasn’t one that just went away as organizations grew past the startup phase, but one that got even more tangled up and unpleasant.

We’ve been building the product for a little over a year now, and it’s been interesting to learn that some problems were surprisingly easy, and others unusually tough. To get the data we need to train our models, we use the same underlying technology that tools like Fullstory and Hotjar use, and it works quite well. Also, we’re able to get good tests from relatively few user sessions (in most cases, fewer than 200 sessions).

Right now, the models are really good at improving test coverage for featureful web-apps that don’t have much coverage (ie; generating and maintaining a bunch of new tests), but making existing tests better has been a tougher nut to crack. We don’t have as much of a place in organizations where test coverage is great and test quality is medium-to-poor, but we’re keen to develop in that direction.

We’re still early, and spend basically all of our time working with a small handful of design partners (mostly medium-sized startups struggling with test coverage), but it felt like time to share with the HN community.

Thanks so much, happy to answer any questions, and excited to hear your thoughts!

Comments URL: https://news.ycombinator.com/item?id=35629050

Points: 78

# Comments: 32