When AI Writes the Tests, What Should People Check?

Click a button on an online store. Place an order. Request a return and check the refund amount.

Checking such operations and their outcomes over and over has been automated for years. Now AI is also being used to prepare test plans—what should be checked and in what order—and to create the programs that carry them out.[1][2]

But a successful test result does not guarantee that the service is safe or comfortable to use. The problems a test can find depend on which conditions were included and what the test was told to consider correct.

In DoNext!! Interview 04, Misaki Ogawa, a new employee living in Fukuoka in the fictional year 2035, examines order and return conditions that catch her attention in AI-generated test results. She checks quality alongside a senior colleague. It's a scene from an imagined future.

How much of that work is already beginning to happen?

From Recording Actions to Asking for a Test Plan

One tool for automating checks of web services is Playwright. It is a developer tool that controls a browser, enters information, clicks elements, and verifies the results.

Its recording feature can turn a person's actions in the browser into test code. Checks such as "this text appears" or "this field contains the expected value" can also be added. This feature is based on recorded user actions and should be distinguished from judgments made by generative AI.[1]

Playwright's AI-powered Test Agents include a planner that explores an application and prepares a plan, and a generator that turns the plan into test code. Instead of recording every step manually first, developers can begin by describing what they want to verify.[2]

For example, imagine asking, "Can a customer buy something without creating an account?" The request could be broken down into choosing a product, entering an address, paying, and checking that the order is complete. This is an illustrative use case imagined by the editorial team, not the result of hands-on testing.

The person responsible can review the proposed plan and add missing conditions. Beyond a successful purchase, they might want to check what happens if the address is blank or an item goes out of stock.

Describe the Goal in Ordinary Language, Then Build the Test

A similar shift is visible in commercial testing services.

According to mabl's official documentation, users can describe what they want to check in everyday language, and AI can generate a web application's test structure and actions. Alongside existing tests, users can refer to requirements documents or images of expected screens. If the proposed structure misses the goal, they can revise it with additional instructions before generation continues.[3]

If you simply say, "Check returns," the result might be a straightforward sequence in which a return goes through successfully. Add conditions such as "What if the customer returns just one of three items?" or "What if they used a coupon?" and the real question becomes clearer.

These return examples are not results from tests of a particular product. They illustrate how to describe a problem to AI. This is where an employee's knowledge of everyday business operations can become a set of specific test conditions.

A Working Button Doesn't Always Mean a Usable Screen

The page might say an order is complete while the amount underneath overlaps another element and cannot be read. Everything might look fine on a desktop, but on a phone the return button could be impossible to find.

One way to find changes that are hard to detect by checking actions and text alone is to compare the screens themselves. Applitools Eyes is a product for AI-assisted visual testing, including checks for overlapping elements and shifts in layout.[4]

The basic approach is to compare a new screen against an accepted baseline. When a difference appears, someone decides whether it is an intended change or a defect. An approved design change becomes the new baseline; for a defect, the original baseline is kept.[5]

There is judgment involved here, too. If a screen has lost its explanation of refund conditions and someone accepts it as the new baseline, future comparisons will use a version without that explanation. This is a caution drawn from how visual comparison works, not a report of a defect in any particular product.

The ability to identify a visual difference is closely linked to the decision about whether that difference should be accepted.

What Does "Automatically Fixing" a Test Actually Fix?

When a service's interface changes, an older test may no longer find the element it was supposed to operate. The purchasing function can still work perfectly even though the test itself needs to be updated.

mabl describes an auto-healing feature that looks for an element with similar characteristics when the previous locator fails. Its documentation also notes that a mistaken match can produce false successes or failures.[6]

Playwright's healer similarly retries failed steps, inspects the screen, and attempts to repair the test. Its documentation also describes outcomes where a test is skipped if the functionality is judged to be broken.[2]

Automatic test repair does not mean that all defects in an application have been fixed.

For example, replacing "The refund amount should be ¥1,000" with "The refund amount field is displayed" might make a test easier to pass. But it would no longer verify the amount itself. This is an example of weakening an assertion, not a claim that any of these products necessarily make that particular change.

Alongside the new pass/fail result, we need to see what was rewritten and what is no longer being checked.

Tracing the Result Back and Explaining Why

Test automation tools also provide records that help people investigate results.

Playwright's Trace Viewer allows developers to examine screen states before and after actions, errors, and network activity. It provides clues about what was entered and how the application responded, rather than showing only where a test failed.[7]

Some of today's tools already point toward the fictional future where Misaki discusses order conditions with a senior colleague. An inexperienced employee might look beyond a green test result, inspect the same record, and ask, "Why did we need to check this condition?"

Of course, a detailed trace does not automatically explain the business rules behind a transaction. People building the service still have to decide how refund amounts are calculated and what customers need to be told.

More automated tests do not necessarily mean that every problem a customer might face will be found. Even Playwright's documentation on accessibility testing explains that automated checks have limits and should be combined with manual checks and testing with users.[8]

Once the Repetitive Work Is Delegated, What Questions Can We Ask?

AI is already becoming involved in preparing tests and adapting them to interface changes. It may make it possible to check more conditions that teams previously set aside because of the work required to prepare the tests.

Even with an ordinary shopping site, we can go beyond "Could the customer place an order?" and ask, "Could they cancel an order made by mistake?" or "Was the explanation of the refund amount understandable?"

People who know shopping, customer service, and day-to-day operations could help express the right conditions to check, discuss the AI-generated steps and results, and explore where users might struggle. That creates ways to contribute beyond writing test code alone.

Misaki's 2035 is a fictional portrayal of that kind of work and learning as part of someone's everyday life. None of the material reviewed here establishes how many people will be hired in the future or whether everyone will have the option of working this way.

Still, tools already exist that bring us closer to delegating repetitive operations and making time to discuss troubling results with an experienced colleague. The important question may be how people use those tools, guided by what they actually want to verify. That is one possible direction for the future of software quality assurance.