Flaky Tests: Why They Happen and How to Actually Fix Them
Flaky tests flip because they lean on timing, markup, shared data or machine load. See where each dependency hides and why retries never take it out.
In a 2024 study of 52 software projects, 46.5% of flaky tests passed or failed depending on how much CPU and memory the run was given. Nobody touched the code between those runs. The machine changed, and the tests had been leaning on it all along.
Nearly every flaky test works this way. It depends on something nobody meant it to, like a timing guess, a selector pinned to markup, or a row another test also writes. Retries and longer timeouts leave that dependency in place. Take it out, and the flake goes with it.
What you’ll learn
- What makes a test flaky, and how flaky differs from brittle
- The four things flaky tests depend on, and why they fail in clusters
- Why retries, longer timeouts and quarantine leave the cause in place
- What vision-based testing removes from a suite, and what it leaves to you
What Is a Flaky Test
A flaky test gives different results on the same code. Run it ten times against one commit and it passes eight and fails two, with nothing in the diff to explain the difference. Something the test depends on varies between runs, and the result varies with it, which is the textbook case of non-deterministic tests. Take a checkout test that reads the cart total right after clicking through:
// Passes about 8 runs in 10
test('checkout completes successfully', async ({ page }) => {
await page.click('#add-to-cart');
await page.click('#checkout-button');
// Sometimes the cart total hasn't updated yet
expect(await page.textContent('.total')).toBe('$99.00');
});
The cart updates asynchronously. Sometimes the new total lands before the assertion reads it, sometimes after, and the test reads it exactly once. Nothing in the test logic is wrong. What it depends on, how fast the cart service answers, is outside its control.
Two different failures get filed under “flaky.” A brittle test fails on every run after a harmless change, such as a renamed CSS class, and keeps failing until someone repairs it. A flaky test flips on unchanged code. Rerunning the failing test on the same commit tells you which you have, though a flip can occasionally come from a real race condition in your product, the kind a user can hit too.
Four Things Flaky Tests Depend On
Research taxonomies split the root causes of flakiness more finely, into async waits, concurrency, test order and more. Each of those reduces to one of four dependencies a test was never meant to have.
- Timing: The test assumes an operation finishes inside a fixed window. A slow query or a busy CI runner breaks the assumption, and the resource-affected tests in that 2024 study sit here.
- Locators: A locator built on generated class names or list positions can match a different element, or none, from one run to the next. The same coupling turns a harmless refactor into a brittle failure.
- Environment: Browser versions, screen sizes, network routes and runner hardware differ between a laptop and CI, so a test can pass on one and flip on the other.
- Shared state: A test that leaves data behind changes the next test’s starting point, and suddenly test order matters. Without isolating tests, one test’s leftovers become another test’s failure. Pagination tests are a clear case: an assertion that names the rows on page two goes red the moment another test inserts a record, and a pagination test built on invariants does not.
These dependencies rarely surface one test at a time. When Owain Parry and colleagues analyzed 10,000 test-suite runs across 24 Java projects, about three-quarters of the 810 flaky tests belonged to clusters that failed together, and the clusters averaged 13.5 tests. Most traced back to intermittent networking and unstable external dependencies. So when several tests fail in the same run, look for the dependency they share.
Why Shipping Faster Makes Flaky Tests Worse
Every merge is a chance for something a test depends on to shift, and merges across GitHub jumped last year. Developers merged 43.2 million pull requests a month in 2025, up 23% from 35 million the year before, and GitHub notes the acceleration coincided with the preview of its Copilot coding agent.
Coding assistants push that pace further, and they have no idea which class names or response times your tests assume. A routine change can break a test outright, or turn a stable one flaky:
| Routine change | What the test was relying on | What you see |
|---|---|---|
| A CSS class renamed in a refactor | The selector .btn-primary | Fails every run until someone repairs it |
| A new API call added before the cart renders | The total being ready when the assertion reads it | Passes on fast runs, fails on slow ones |
| A query loses its ORDER BY | Row order, through an nth-child(1) locator | Clicks a different row whenever the database returns rows in a different order |
| CI moves to smaller runners | Enough CPU to finish inside a timeout | Flips more often under load |
Only the first row is a broken test in the usual sense, and it announces itself. The other three get through code review and the author’s machine, then start flipping in CI later. When a stable test starts flipping, read the merges from the day it started before you read the test.
The Hidden Costs of Ignoring Flaky Tests
Hitting rerun takes a minute. The rest of the bill never reaches a sprint board, and it lands in four places:
- Compute waste: A 20-minute pipeline rerun twice a day costs each developer 40 minutes of waiting. Across a team of ten, that’s 400 minutes, close to 7 hours of CI time a day spent proving nothing broke.
- Context switching: Every flaky failure pulls someone out of their work to chase a failure that turns out to be noise. The context they had built doesn’t come back for free. One mobile team told us what they disliked about their old testing tool: tests that flaked, and a fix that could mean waiting up to half an hour for the rerun.
- Trust erosion: Once failures look random, people stop reading them. A real regression arrives looking exactly like the last ten false alarms and gets rerun past, which defeats the point of regression testing.
- Morale: Strong engineers resent debugging tests that were never broken, and the resentment lands on the suite.
My co-founder Jinoo has a sharper breakdown of what flaky tests really cost, including why a rerun that’s cheap per click gets expensive per quarter. Put your own numbers on the compute line first. Finance will believe that one.
Why Common Fixes Backfire
Faced with a red build and a deadline, teams reach for four standard responses. All four go after the flaky test itself, and none of them reaches the dependency underneath.
- Retry logic: Rerun failures until they pass. The retry hides the flake without touching its cause, and the odds stack up as the suite grows. Give 1,000 tests a 0.1% flake rate each and about 63% of full runs hit at least one flaky failure, because a clean run needs all 1,000 to pass and 0.999 to the 1,000th power is about 0.37.
- Longer timeouts: When a test already waits for a condition, extra headroom can help a step that is slow but reliable. When it sleeps for a fixed time, a longer sleep moves the guess and adds its full length to every run, pass or fail.
- Quarantine without a deadline: Pulling the worst offenders off the critical path works when each test has someone accountable and a time limit, the setup our guide on how to fix flaky tests walks through. Without them the quarantine only grows, and coverage shrinks without anyone deciding it should. Ask a team with a big UI suite what share of it is still active. A lot of those tests end up commented out over flakiness.
- Maintenance sprints: Rotate engineers onto test repair for a sprint. They fix a dozen flaky tests, and the next month’s merges create another dozen, because nothing changed about how the tests attach to the app.
Look, retrying a flaky test is turning up the radio over an engine knock. The noise goes away and the engine keeps knocking. Find out what the test is coupled to. The retry setting can wait.
Test What Users See
Bring a staging URL. See tests run with no selectors to write.
Why Traditional Frameworks Create Flakiness
Selector-based frameworks tie a test to the page’s code structure, and code structure is the part of the page developers change most freely. Selenium tests usually find elements by CSS selector or XPath, and Cypress tests by CSS selector or a data attribute. The test never asks whether a person can see the checkout button. It checks whether div.container > button.primary-cta exists.
Playwright’s own API docs steer you away from CSS, toward roles, accessible names and test IDs, which survive a renamed class. It ships defenses against flaky Playwright tests too, auto-waiting among them. Its locators resolve against the DOM and the accessibility tree, though, so a restructured component or a reworded label can break them:
// Breaks when someone renames the class or moves the wrapper
await page.locator('div.container > button.primary-cta').click();
// Survives a renamed class, breaks when the role or label changes
await page.getByRole('button', { name: 'Checkout' }).click();
A refactor or a library upgrade can move the markup under a test. The button looks the same to a user, and the selector misses anyway.
Explicit waits fix the timing dependency, and they’re worth writing. Sturdier locators shrink the locator dependency without removing it, because the test keeps pointing at structure a developer is free to change.
Check the locator on any UI test that goes red right after a refactor.
What Vision-Based Testing Removes, and What It Leaves
If coupling to markup is what breaks, the durable fix is a test that doesn’t read markup to find things. Vision-based testing locates elements from what’s rendered on screen, the way a person scanning the page would. Pie works vision-first. You write a step like “add a product to the cart,” and Pie finds the button on the rendered screen, so nobody on your team writes or maintains button#add-to-cart.primary. Three dependencies drop out when the test reads the screen instead of the markup:
- Selector upkeep: A renamed class or a restructured component can come through with no test edits.
- Fixed sleeps: Each step checks the screen, so a step written as “wait for the order confirmation” replaces a
sleep(3000)guess. - Moved elements: When a button moves from the sidebar to the header, Pie can still identify it from context, position and function, which the docs on contextual self-healing walk through.
Vision doesn’t reach the other dependencies. Shared data and test order live in your test setup, a starved CI runner starves any test, and a race in your product belongs in the product’s bug tracker. Pie’s self-healing tests take the selector work off your team’s list.
| Aspect | Selector-Based | Vision-Based |
|---|---|---|
| How elements are found | CSS, XPath, roles, test IDs | What’s rendered on screen |
| Class rename or refactor | CSS and XPath locators need repair | Can need no change |
| Who maintains locators | Your team | No locators for your team to maintain |
| Shared state and test order | Your team’s job | Your team’s job |
Take the Dependency Out of the Test
Each standard fix, from the retry to the quarantine, treats the flaky test as the thing to manage. The thing to remove is what it depends on, whether that’s the runner’s CPU or a class name.
Most of that removal is ordinary engineering. Wait for the state the test needs, give every test its own data, and stop pinning tests to markup that changes every sprint.
We built Pie to take markup out of the test. Your team writes plain-English steps and Pie carries them out vision-first, reading the screen. For every flow it covers, selector upkeep drops off your team’s list. Your shared data is yours to fix, and once it is, the failures left are worth reading.
See It on Your Own App
Pick a flow that flakes today. Watch Pie run it from the screen.
Frequently Asked Questions
It can. Your team writes Pie tests as plain-English steps without maintaining selectors, and Pie works vision-first from the rendered screen, so a renamed class or a moved button can need no repair.