Blog / Flaky Tests: Why They Happen and How to Actually Fix Them
Guide

Flaky Tests: Why They Happen and How to Actually Fix Them

Flaky tests flip because they lean on timing, markup, shared data or machine load. See where each dependency hides and why retries never take it out.

In a 2024 study of 52 software projects, 46.5% of flaky tests passed or failed depending on how much CPU and memory the run was given. Nobody touched the code between those runs. The machine changed, and the tests had been leaning on it all along.

Nearly every flaky test works this way. It depends on something nobody meant it to, like a timing guess, a selector pinned to markup, or a row another test also writes. Retries and longer timeouts leave that dependency in place. Take it out, and the flake goes with it.

What you’ll learn

  • What makes a test flaky, and how flaky differs from brittle
  • The four things flaky tests depend on, and why they fail in clusters
  • Why retries, longer timeouts and quarantine leave the cause in place
  • What vision-based testing removes from a suite, and what it leaves to you

What Is a Flaky Test

A flaky test gives different results on the same code. Run it ten times against one commit and it passes eight and fails two, with nothing in the diff to explain the difference. Something the test depends on varies between runs, and the result varies with it, which is the textbook case of non-deterministic tests. Take a checkout test that reads the cart total right after clicking through:

// Passes about 8 runs in 10
test('checkout completes successfully', async ({ page }) => {
  await page.click('#add-to-cart');
  await page.click('#checkout-button');
  // Sometimes the cart total hasn't updated yet
  expect(await page.textContent('.total')).toBe('$99.00');
});

The cart updates asynchronously. Sometimes the new total lands before the assertion reads it, sometimes after, and the test reads it exactly once. Nothing in the test logic is wrong. What it depends on, how fast the cart service answers, is outside its control.

Two different failures get filed under “flaky.” A brittle test fails on every run after a harmless change, such as a renamed CSS class, and keeps failing until someone repairs it. A flaky test flips on unchanged code. Rerunning the failing test on the same commit tells you which you have, though a flip can occasionally come from a real race condition in your product, the kind a user can hit too.

Four Things Flaky Tests Depend On

Research taxonomies split the root causes of flakiness more finely, into async waits, concurrency, test order and more. Each of those reduces to one of four dependencies a test was never meant to have.

  1. Timing: The test assumes an operation finishes inside a fixed window. A slow query or a busy CI runner breaks the assumption, and the resource-affected tests in that 2024 study sit here.
  2. Locators: A locator built on generated class names or list positions can match a different element, or none, from one run to the next. The same coupling turns a harmless refactor into a brittle failure.
  3. Environment: Browser versions, screen sizes, network routes and runner hardware differ between a laptop and CI, so a test can pass on one and flip on the other.
  4. Shared state: A test that leaves data behind changes the next test’s starting point, and suddenly test order matters. Without isolating tests, one test’s leftovers become another test’s failure. Pagination tests are a clear case: an assertion that names the rows on page two goes red the moment another test inserts a record, and a pagination test built on invariants does not.

These dependencies rarely surface one test at a time. When Owain Parry and colleagues analyzed 10,000 test-suite runs across 24 Java projects, about three-quarters of the 810 flaky tests belonged to clusters that failed together, and the clusters averaged 13.5 tests. Most traced back to intermittent networking and unstable external dependencies. So when several tests fail in the same run, look for the dependency they share.

Why Shipping Faster Makes Flaky Tests Worse

Every merge is a chance for something a test depends on to shift, and merges across GitHub jumped last year. Developers merged 43.2 million pull requests a month in 2025, up 23% from 35 million the year before, and GitHub notes the acceleration coincided with the preview of its Copilot coding agent.

Coding assistants push that pace further, and they have no idea which class names or response times your tests assume. A routine change can break a test outright, or turn a stable one flaky:

Routine changeWhat the test was relying onWhat you see
A CSS class renamed in a refactorThe selector .btn-primaryFails every run until someone repairs it
A new API call added before the cart rendersThe total being ready when the assertion reads itPasses on fast runs, fails on slow ones
A query loses its ORDER BYRow order, through an nth-child(1) locatorClicks a different row whenever the database returns rows in a different order
CI moves to smaller runnersEnough CPU to finish inside a timeoutFlips more often under load

Only the first row is a broken test in the usual sense, and it announces itself. The other three get through code review and the author’s machine, then start flipping in CI later. When a stable test starts flipping, read the merges from the day it started before you read the test.

The Hidden Costs of Ignoring Flaky Tests

Hitting rerun takes a minute. The rest of the bill never reaches a sprint board, and it lands in four places:

  • Compute waste: A 20-minute pipeline rerun twice a day costs each developer 40 minutes of waiting. Across a team of ten, that’s 400 minutes, close to 7 hours of CI time a day spent proving nothing broke.
  • Context switching: Every flaky failure pulls someone out of their work to chase a failure that turns out to be noise. The context they had built doesn’t come back for free. One mobile team told us what they disliked about their old testing tool: tests that flaked, and a fix that could mean waiting up to half an hour for the rerun.
  • Trust erosion: Once failures look random, people stop reading them. A real regression arrives looking exactly like the last ten false alarms and gets rerun past, which defeats the point of regression testing.
  • Morale: Strong engineers resent debugging tests that were never broken, and the resentment lands on the suite.

My co-founder Jinoo has a sharper breakdown of what flaky tests really cost, including why a rerun that’s cheap per click gets expensive per quarter. Put your own numbers on the compute line first. Finance will believe that one.

Why Common Fixes Backfire

Faced with a red build and a deadline, teams reach for four standard responses. All four go after the flaky test itself, and none of them reaches the dependency underneath.

  1. Retry logic: Rerun failures until they pass. The retry hides the flake without touching its cause, and the odds stack up as the suite grows. Give 1,000 tests a 0.1% flake rate each and about 63% of full runs hit at least one flaky failure, because a clean run needs all 1,000 to pass and 0.999 to the 1,000th power is about 0.37.
  2. Longer timeouts: When a test already waits for a condition, extra headroom can help a step that is slow but reliable. When it sleeps for a fixed time, a longer sleep moves the guess and adds its full length to every run, pass or fail.
  3. Quarantine without a deadline: Pulling the worst offenders off the critical path works when each test has someone accountable and a time limit, the setup our guide on how to fix flaky tests walks through. Without them the quarantine only grows, and coverage shrinks without anyone deciding it should. Ask a team with a big UI suite what share of it is still active. A lot of those tests end up commented out over flakiness.
  4. Maintenance sprints: Rotate engineers onto test repair for a sprint. They fix a dozen flaky tests, and the next month’s merges create another dozen, because nothing changed about how the tests attach to the app.

Look, retrying a flaky test is turning up the radio over an engine knock. The noise goes away and the engine keeps knocking. Find out what the test is coupled to. The retry setting can wait.

Test What Users See

Bring a staging URL. See tests run with no selectors to write.

Why Traditional Frameworks Create Flakiness

Selector-based frameworks tie a test to the page’s code structure, and code structure is the part of the page developers change most freely. Selenium tests usually find elements by CSS selector or XPath, and Cypress tests by CSS selector or a data attribute. The test never asks whether a person can see the checkout button. It checks whether div.container > button.primary-cta exists.

Playwright’s own API docs steer you away from CSS, toward roles, accessible names and test IDs, which survive a renamed class. It ships defenses against flaky Playwright tests too, auto-waiting among them. Its locators resolve against the DOM and the accessibility tree, though, so a restructured component or a reworded label can break them:

// Breaks when someone renames the class or moves the wrapper
await page.locator('div.container > button.primary-cta').click();

// Survives a renamed class, breaks when the role or label changes
await page.getByRole('button', { name: 'Checkout' }).click();

A refactor or a library upgrade can move the markup under a test. The button looks the same to a user, and the selector misses anyway.

Explicit waits fix the timing dependency, and they’re worth writing. Sturdier locators shrink the locator dependency without removing it, because the test keeps pointing at structure a developer is free to change.

Check the locator on any UI test that goes red right after a refactor.

What Vision-Based Testing Removes, and What It Leaves

If coupling to markup is what breaks, the durable fix is a test that doesn’t read markup to find things. Vision-based testing locates elements from what’s rendered on screen, the way a person scanning the page would. Pie works vision-first. You write a step like “add a product to the cart,” and Pie finds the button on the rendered screen, so nobody on your team writes or maintains button#add-to-cart.primary. Three dependencies drop out when the test reads the screen instead of the markup:

  • Selector upkeep: A renamed class or a restructured component can come through with no test edits.
  • Fixed sleeps: Each step checks the screen, so a step written as “wait for the order confirmation” replaces a sleep(3000) guess.
  • Moved elements: When a button moves from the sidebar to the header, Pie can still identify it from context, position and function, which the docs on contextual self-healing walk through.

Vision doesn’t reach the other dependencies. Shared data and test order live in your test setup, a starved CI runner starves any test, and a race in your product belongs in the product’s bug tracker. Pie’s self-healing tests take the selector work off your team’s list.

AspectSelector-BasedVision-Based
How elements are foundCSS, XPath, roles, test IDsWhat’s rendered on screen
Class rename or refactorCSS and XPath locators need repairCan need no change
Who maintains locatorsYour teamNo locators for your team to maintain
Shared state and test orderYour team’s jobYour team’s job

Take the Dependency Out of the Test

Each standard fix, from the retry to the quarantine, treats the flaky test as the thing to manage. The thing to remove is what it depends on, whether that’s the runner’s CPU or a class name.

Most of that removal is ordinary engineering. Wait for the state the test needs, give every test its own data, and stop pinning tests to markup that changes every sprint.

We built Pie to take markup out of the test. Your team writes plain-English steps and Pie carries them out vision-first, reading the screen. For every flow it covers, selector upkeep drops off your team’s list. Your shared data is yours to fix, and once it is, the failures left are worth reading.

See It on Your Own App

Pick a flow that flakes today. Watch Pie run it from the screen.

Frequently Asked Questions

A flaky test depends on something that changes between runs, such as response time, which element a locator matches, the machine, or data another test left behind. In a 2024 study of 52 projects, 46.5% of the flaky tests passed or failed depending on the CPU and memory available.
A flaky test flips between pass and fail on unchanged code. A brittle test fails on every run after a harmless change, like a renamed CSS class, until someone repairs it. Both come from coupling a test to something incidental, so the cure for either starts with finding that coupling.
A retry reruns the test without touching what it depends on, so the flake returns on a later run. The odds also compound. With 1,000 tests that each flake 0.1% of the time, about 63% of full runs hit at least one flaky failure.
Many share one cause. A 2025 study of 24 Java projects found about three-quarters of flaky tests belonged to clusters that failed together, mostly traced to intermittent networking and unstable external dependencies. Fixing the shared dependency can calm a whole cluster at once.
Only when the test already waits for a condition and the step is slow but reliable. Lengthening a fixed sleep just moves the guess, and every run pays the extra seconds. Point the wait at the state the test needs, like the cart total showing the new price.
Each merge can shift something a test depends on, like a class name, the order of a list, or a new network call before a screen renders. GitHub counted 43.2 million merged pull requests a month in 2025, up 23%, so those shifts arrive more often.

It can. Your team writes Pie tests as plain-English steps without maintaining selectors, and Pie works vision-first from the rendered screen, so a renamed class or a moved button can need no repair.

No, not to run tests. You give Pie the app URL or build, working test credentials and any setup steps. Connecting a repository is optional. You only need one for Pie Loop, which tests your pull requests and can prepare proposed fix pull requests for your team to review.
Dhaval Shreyas
Dhaval Shreyas
CEO & Co-founder at Pie

13 years building mobile infrastructure at Square, Facebook, and Instacart. Now building the QA platform he wished existed the whole time. LinkedIn →