Guide

How to Fix Flaky Tests: Detect Them, Fix the Root Cause, Stop Making More

Flaky tests are multiplying in the AI-code era. The 2026 playbook: detect them before they block CI, fix the async, state, and order root causes, and stop generating new ones.

Dhaval Shreyas

CEO & Co-founder at Pie

12 min read Updated Jul 16, 2026

Why Flaky Tests Are Multiplying in 2026

Flaky tests are multiplying for two reasons, and both trace back to the same source: your team is shipping far more code, and a growing share of it is written by machines that default to the exact patterns that cause flakiness. Every other flaky-test guide skips this framing, which is why your flake count keeps climbing no matter how many individual tests you fix.

Reason 1: More Code Means More Tests

Microsoft CEO Satya Nadella said at LlamaCon in April 2025 that 20% to 30% of the company’s code is now written by AI. More code means more tests, and more tests mean more surface area for flakiness. Volume alone lifts your absolute flaky count even when the flake rate holds steady.

Reason 2: AI Writes the Exact Patterns That Flake

The raw count is only half the story. The other half is what AI-generated tests actually look like. A large language model asked to write a test reaches for the patterns that appear most in its training data:

• sleep(2000) instead of a real wait condition
• a brittle CSS selector instead of a stable one
• a shared fixture instead of hermetic data

Those are among the most common triggers of flaky tests, generated at scale, by default. An Uplevel Data Labs study of nearly 800 developers found bug rates rose 41% among developers with GitHub Copilot access, compared to developers without it. The tests those assistants write inherit the same instability, a dynamic we unpack in our guide to testing AI-generated code.

The practical consequence: fixing flaky tests one at a time is now a losing race against your own commit velocity. You still fix the ones in front of you, and the rest of this guide shows you how. But you also have to change how tests get created so the flaky ones stop arriving in the first place.

Detect Them Before You Fix Them

You cannot fix, triage, or prevent a flaky test until you can prove which tests are flaky, so detection comes before every other step. The most common mistake teams make is starting from memory, arguing about which tests “feel unreliable” in standup instead of pulling the numbers. Detection replaces the argument with data.

The baseline detection method needs no special tooling: rerun the suite. A test that passes and fails across repeated runs on the same commit is flaky by definition. Run the full suite, or the changed tests, several times and record which ones flip.

# Run the same tests 10 times, record pass/fail each run
for i in {1..10}; do
  npm test -- --changed 2>&1 | tee "run-$i.log"
done

Rerunning gets you started, but it does not scale to thousands of tests. At scale you need the failure history tracked automatically. Slack built exactly that and described it in their flaky-test auto-detection writeup: the system watches every run, flags tests that fail non-deterministically, and suppresses them automatically.

Most CI platforms now surface a per-test flaky rate natively. Cypress Cloud detects and flags flaky tests from your recorded runs when retries are enabled, and test-observability tools do the same across frameworks. If your platform exposes this, turn it on before you write a single fix. Our guide to test observability covers what that data should include.

Detection produces the one input the rest of this guide depends on: a ranked list of your actually-flaky tests, with a failure rate attached to each. Triage picks up from there.

Triage: What to Fix First

Detection hands you the list of flaky tests. Triage decides the order you attack it. Fixing them all with equal urgency is a recipe for burnout, so rank by blast radius. The tests blocking your checkout flow deserve immediate attention. The tests failing intermittently on a deprecated admin page can wait.

Severity Matrix

Add the Prioritization Data

Detection gave you a failure rate per test. Triage needs three more attributes on top of it before you can rank the backlog:

• Failure rate — failures / total runs over 30 days (from detection)
• Last code change — when was the test or its target code modified?
• Coverage scope — what user flows does this test protect?
• Fix complexity — estimated hours based on root cause

Without this data, you’re guessing. Most CI platforms provide historical failure rates. If yours doesn’t, instrument it.

Fixing Async Wait Issues

At Square, the flaky test I hit most often made one mistake: it guessed how long to wait instead of checking whether the thing it needed had actually happened. Async waits are the most common source of flaky tests, and the fix is the same every time.

This is the most-studied category of flakiness, and a decade of research keeps landing on the same number. Luo and colleagues’ foundational “An Empirical Analysis of Flaky Tests” (FSE 2014) put async-wait problems at roughly 45% of all flakiness, ahead of concurrency (20%) and test-order dependency (12%). A 2021 ICSE analysis of 235 flaky UI tests independently reached the same 45%, and industry benchmarks still report it there in 2026. The fix is always the same pattern: replace implicit timing with explicit conditions.

The Problem Pattern

// Flaky: assumes 2 seconds is enough
await page.click('#submit');
await sleep(2000);
const text = await page.textContent('#result');
expect(text).toBe('Order confirmed');

The 2-second guess fails when the server response takes 2.5 seconds under load, an animation delays element visibility, or network latency varies between environments. The wait was never tied to the thing it needed to wait for.

The Fix Pattern

Instead of guessing how long to wait, tell the test exactly what condition must be true before proceeding. The waitForSelector call below doesn’t continue until the result element is actually visible in the DOM. No timing assumptions, no guessing. The test waits for reality to match expectations.

// Stable: waits for actual condition
await page.click('#submit');
await page.waitForSelector('#result', { state: 'visible' });
const text = await page.textContent('#result');
expect(text).toBe('Order confirmed');

Advanced: Compound Conditions

Some operations require multiple things to happen in sequence. A form submission might need both the API response to return AND the UI to update. Using Promise.all lets you wait for the network response while the click triggers the request. The test only proceeds once both the API call completes and the confirmation text appears.

// Wait for loading to complete AND element to appear
await Promise.all([
  page.waitForResponse(resp => resp.url().includes('/api/orders')),
  page.click('#submit')
]);
await page.waitForSelector('#result:has-text("confirmed")');

Framework-Specific Patterns

Each test framework has its own idioms for stable async handling. For Playwright users dealing with more complex flakiness scenarios, we’ve written a dedicated Playwright flaky tests guide.

Playwright builds assertions directly into its expect API. The toHaveText assertion automatically retries until the condition is met or the timeout expires. No explicit wait call needed.

await expect(page.locator('#result')).toHaveText('Order confirmed', {
  timeout: 10000
});

Cypress takes a similar approach with its built-in retry mechanism. The should assertion keeps checking until the element contains the expected text or 10 seconds pass.

cy.get('#result', { timeout: 10000 }).should('contain', 'Order confirmed');

Selenium requires more explicit wait construction through WebDriverWait. You define the condition using Expected Conditions, and the driver polls until that condition returns true.

WebDriverWait(driver, 10).until(
    EC.text_to_be_present_in_element((By.ID, 'result'), 'Order confirmed')
)

Eliminating Shared State

Shared state is the flakiness I trust least, because it sails through code review and only detonates in CI. When Test A writes to a database and Test B reads from it, you’ve created an invisible dependency. Test B passes when it runs after A, fails when it runs before. Multiply this across hundreds of tests and you get a suite where order matters more than code quality. The path out is complete test isolation.

Database Isolation

Option 1: Transaction Rollback

Wrap each test in a transaction, rollback after. Every INSERT, UPDATE, and DELETE gets undone when the test finishes. The next test starts with a pristine database.

@pytest.fixture(autouse=True)
def transactional_test(db_connection):
    transaction = db_connection.begin()
    yield
    transaction.rollback()

Rollback is fast because there’s no actual data to clean up, so the reset is instant. The limitation shows up when your test code opens multiple database connections, since each connection has its own transaction scope.

Option 2: Unique Data Per Test

Generate unique identifiers for all test data. Each test creates its own users, orders, and records with UUIDs or timestamps baked into the identifiers. Tests never collide because they’re operating on completely separate data.

const testId = `test_${Date.now()}_${Math.random().toString(36).substr(2, 9)}`;
const user = await createUser({ email: `${testId}@test.com` });

Unique test data works with any architecture, including microservices and distributed systems. The tradeoff is that test data accumulates over time, so you’ll need a periodic cleanup job to purge old test records, or your test database grows indefinitely.

Option 3: Dedicated Database Per Test Suite

Spin up isolated databases for parallel test execution. Each test worker gets its own Postgres instance. Zero possibility of cross-contamination.

# docker-compose.test.yml
services:
  db-suite-1:
    image: postgres:14
    environment:
      POSTGRES_DB: test_suite_1
  db-suite-2:
    image: postgres:14
    environment:
      POSTGRES_DB: test_suite_2

True isolation comes at a cost. You need infrastructure to spin up and tear down databases, and each instance takes time to initialize. For large test suites running in parallel, the isolation is worth the setup overhead.

Cache Isolation

Redis, Memcached, in-memory caches. They all persist data between test runs unless you explicitly clear them. A test that passes locally fails in CI because someone else’s test populated the cache with stale data.

Option 1: Flush before each test

The nuclear option. Wipe everything before each test starts. Simple and reliable, but slow if your cache is large.

beforeEach(async () => {
  await redis.flushall();
  await memcached.flush();
});

Option 2: Test-specific key prefixes

Each test worker gets its own namespace. Tests running in parallel never read each other’s cached values because they’re looking at different keys entirely.

const cachePrefix = `test_${process.env.TEST_WORKER_ID}_`;
cache.get(`${cachePrefix}${key}`);

Prefixing is faster than flushing because you’re not clearing data that doesn’t affect your test. The tradeoff is that you need to consistently apply the prefix across all cache operations in your test code.

Browser State Isolation

Login sessions, shopping carts, form data. Browser state accumulates across tests unless you explicitly reset it. A test that expects a logged-out state fails because a previous test logged in and never logged out.

Creating a fresh browser context for each test gives you a clean slate. No cookies, no localStorage, no session data carried over from previous tests. The newContext() call below creates an isolated browser environment that behaves like a fresh incognito window.

// Playwright
test.beforeEach(async ({ browser }) => {
  const context = await browser.newContext();
  // Use this context for the test
});

External Dependency Isolation

Third-party APIs are shared state you don’t control, and they hit API and integration suites hardest. It’s the reason a Rest Assured or Supertest suite that passes at 9am fails at 2pm. A real payment gateway, geolocation service, or partner API rate-limits you, times out, or returns a slightly different payload. The fix is to stop calling the real thing in your test lane. Mock or stub every external dependency so the response is deterministic, and reserve live-integration checks for a separate, non-blocking contract-test job.

// Stub the flaky third-party call, assert on your own logic
nock('https://api.partner.com')
  .get('/v1/rates')
  .reply(200, { usd: 1.0 });

Keep one honest exception. Run a nightly job that hits the real endpoints and alerts when the contract drifts. That’s where you want the partner’s flakiness surfaced, not scattered across every PR build.

Breaking Order Dependencies

Run your test suite twice with different orderings. If results differ, you have order dependencies. These are some of the trickiest flaky tests to debug because they only surface when test execution order changes, which happens unpredictably in parallel CI runs.

Detection

Order dependencies surface frequently in suites that grew organically without strict isolation. The tests passed for years because they happened to run in alphabetical order. Then someone adds parallelization and suddenly half the suite fails.

Randomizing test execution order forces these hidden dependencies to surface. When Test B fails only when it runs before Test A, you’ve found an order dependency. The flags below tell each framework to shuffle the test order using a seed value you can reproduce later for debugging.

# Pytest
pytest --randomly-seed=12345

# Jest
jest --randomize

# RSpec
rspec --order random

Run your full suite three times with different seeds. If any test fails inconsistently, investigate whether it depends on state created by another test.

Common Patterns and Fixes

Pattern 1: Test A creates data, Test B expects it

It’s the most common order dependency. Test A creates a database record. Test B queries for that exact record by ID. When A runs first, B passes. When B runs first, it fails because the record doesn’t exist yet. The fix is making each test responsible for creating and cleaning up its own data.

// Before (dependent)
test('A: create user', async () => {
  await createUser({ id: 1, name: 'Test' });
});

test('B: fetch user', async () => {
  const user = await getUser(1); // Assumes A ran first
  expect(user.name).toBe('Test');
});

// After (independent)
test('A: create user', async () => {
  await createUser({ id: 1, name: 'Test' });
  // Cleanup
  await deleteUser(1);
});

test('B: fetch user', async () => {
  // Create its own data
  await createUser({ id: 2, name: 'Test' });
  const user = await getUser(2);
  expect(user.name).toBe('Test');
  await deleteUser(2);
});

Pattern 2: Global state modification

Feature flags, configuration objects, singleton instances. When one test modifies global state and doesn’t restore it, every subsequent test runs in a polluted environment. The fix is capturing the original state before each test and restoring it afterward, regardless of whether the test passes or fails.

// Before (modifies global)
test('A: enable feature', () => {
  featureFlags.enable('new_checkout');
  // Test with flag enabled
});

test('B: expects default state', () => {
  // Fails if A ran first and didn't reset
  expect(featureFlags.isEnabled('new_checkout')).toBe(false);
});

// After (isolated)
let originalFlags;

beforeEach(() => {
  originalFlags = featureFlags.getAll();
});

afterEach(() => {
  featureFlags.setAll(originalFlags);
});

Retry, Quarantine, or Delete?

Quarantine is the right default for a flaky test you can’t fix today, and it beats the two options teams reach for instead. When a test flakes, you have three choices, and only one of them is honest.

Retry it until it passes, which masks real bugs behind a green checkmark and is the worst default. Quarantine it, which stops it blocking CI while keeping the coverage intent visible. Or delete it, which is correct only after quarantine has proven the test isn’t earning its keep. Engineers who’ve been burned by a suite full of retries tend to land in the same place: quarantine first, fix if the flow matters, delete if it doesn’t.

Quarantine provides the middle ground the other two options skip. Don’t block CI, but don’t silently lose coverage either.

Implementation

Step 1: Tag flaky tests

Mark tests as flaky in your code so tooling can identify them. The test.skip with a flaky flag tells your test runner this test is known to be unreliable. Some teams prefer moving flaky tests to a separate directory entirely, which makes filtering easier in CI.

test.skip('known flaky: checkout flow', { flaky: true }, async () => {
  // Test code
});

The directory approach gives you physical separation. Everything in stable/ runs in the blocking CI job. Everything in quarantine/ runs separately.

tests/
  stable/
  quarantine/

Step 2: Run quarantined tests separately

Configure your CI pipeline to run stable and quarantined tests as separate jobs. The stable job blocks merges on failure. The quarantine job runs the same tests but with allow_failure: true, so flaky failures don’t block deploys while you work on fixes.

# CI pipeline
jobs:
  stable-tests:
    script: npm test -- --ignore 'quarantine/**'

  quarantine-tests:
    script: npm test -- quarantine/**
    allow_failure: true

Step 3: Monitor quarantine metrics

Quarantine without visibility becomes a dumping ground. Track these numbers weekly and display them on a dashboard your team sees daily.

• Number of tests in quarantine — is it growing or shrinking?
• Days in quarantine per test — which tests have been stuck the longest?
• Quarantine entry/exit rate — are you fixing faster than you’re adding?

Step 4: Enforce quarantine limits

Set a policy: tests in quarantine for more than 30 days must be fixed or deleted. No exceptions. Schedule weekly quarantine review where someone is accountable for reviewing each quarantined test and either committing to a fix date or deleting it.

Preventing Future Flakiness

Every flaky test you fix today could have been prevented yesterday. The patterns that produce flakiness are predictable: hardcoded sleeps, shared database records, implicit dependencies on test execution order. Baking prevention into your test architecture costs less than debugging intermittent failures at 2 AM.

Test Design Principles

Principle 1: No implicit waits

Ban sleep() calls at the linter level. The ESLint rules below flag any test that uses an empty waitFor callback or queries elements with getBy instead of findBy. These rules turn implicit timing assumptions into CI failures before they become flaky tests in production.

// eslint-plugin-testing-library enforces this
rules: {
  'testing-library/no-wait-for-empty-callback': 'error',
  'testing-library/prefer-find-by': 'error'
}

Principle 2: Factory functions for data

Every test should create its own data using factory functions that generate unique identifiers. The factory below creates users with UUID-based emails so two tests can never collide on the same record. Overrides let individual tests customize specific fields while keeping everything else random.

// test-utils/factories.js
export function createTestUser(overrides = {}) {
  return {
    id: uuid(),
    email: `${uuid()}@test.example`,
    createdAt: new Date(),
    ...overrides
  };
}

Principle 3: Environment parity

Tests that pass locally but fail in CI usually rely on environmental differences. Running your test database and cache in Docker makes your local environment match CI exactly. The compose file below spins up the same Postgres and Redis versions your CI uses.

services:
  app:
    build: .
    depends_on:
      - db
      - redis
  db:
    image: postgres:14
  redis:
    image: redis:7

CI Pipeline Hardening

These patterns focus on build-time prevention. For comprehensive CI/CD pipeline strategies including quarantine workflows and parallel test execution, see our flaky tests in CI/CD guide.

Run new tests multiple times before merge

Catch flaky tests before they merge by running new or changed tests multiple times in CI. If a test passes once but fails on runs two through five, you’ve caught flakiness before it pollutes your main branch. The workflow below runs only the tests affected by the PR, five times in a row.

on: pull_request

jobs:
  stability-check:
    steps:
      - name: Run new tests 5 times
        run: |
          for i in {1..5}; do
            npm test -- --changed
          done

Block merge on flaky detection

If your test framework outputs a flaky rate metric, gate merges on it. The script below reads the flaky rate from your test results and fails the build if more than 5% of test runs were inconsistent, keeping PRs that introduce new flakiness from merging.

- name: Check flaky rate
  run: |
    if [[ $(cat test-results/flaky-rate.txt) -gt 5 ]]; then
      echo "Flaky rate above 5%, failing build"
      exit 1
    fi

Team Habits That Prevent Flaky Tests

You can have perfect test architecture and still accumulate flaky tests if your team treats them as someone else’s problem. The engineering teams that maintain healthy test suites share common practices. They make flakiness visible, assign clear ownership, and treat test reliability as a first-class concern.

1. The “Boy Scout Rule” for Tests

Leave the test suite cleaner than you found it. If you encounter a flaky test during unrelated work:

• Spend 15 minutes investigating
• If fixable quickly, fix it
• If not, document what you learned and add to the backlog

2. Flaky Test Friday

Dedicate one hour weekly to flaky test triage. Rotate ownership. Track tests fixed, tests quarantined, and tests deleted. Make it a game. Celebrate reductions.

3. Blame-Free Flaky Test Creation

Flaky tests happen. Don’t penalize engineers for creating them. Penalize for leaving them unfixed. The CI failure that identifies flakiness is doing its job.

Measuring Progress

You can’t improve what you don’t measure. Without metrics, flaky test discussions become finger-pointing sessions. With metrics, they become engineering conversations about trade-offs and priorities. Track these monthly and display them where your team can see them.

Put these on a dashboard your team sees every day. When the numbers are visible, they become part of the conversation.

How Pie Keeps Tests From Flaking

Everything above fixes flaky tests you already have. Pie changes what gets written in the first place by attacking the two largest categories at their root.

What Pie Removes by Design

Async waits and selector-based failures are the two biggest buckets of flakiness, and both come from the same place. A test hardcodes how long to wait, or it pins an element to a selector that breaks on the next redesign. Those are exactly the patterns an LLM reaches for when it writes a test, which is why AI-generated suites flake so much.

Pie’s vision-based, self-healing tests recognize elements by what they look like on screen, the way a person would. There is no sleep(2000) to expire and no locator to repair, so async-wait and selector failures never enter the suite. When your test volume is climbing because of AI, removing those two categories by design matters more than any single test you’ll fix by hand this week.

What Pie Leaves to You

Shared state and test-order dependencies live in your own architecture, and the isolation work earlier in this guide is still yours to do. What Pie removes is the flakiness that used to arrive with every AI-written test, before it ever reaches your pipeline.

Build the Infrastructure, Then the Tests

Every flaky test that blocks a deploy costs more than the hour you spend investigating it. It costs the trust that makes continuous deployment work. When engineers stop believing red builds, they start ignoring them, and that’s when real bugs ship.

Explicit waits, isolated state, and honest quarantine get you to single-digit flake rates, the same way they did for Slack and GitHub. But one-at-a-time fixes only solve today’s problem. The durable win is making flakiness structurally hard to introduce, which is the bet we made when we built Pie. Kill the hardcoded waits and brittle selectors at the source, and your suite goes back to being the safety net it was supposed to be.

Dhaval Shreyas

CEO & Co-founder at Pie

13 years building mobile infrastructure at Square, Facebook, and Instacart. Now building the QA platform he wished existed the whole time. LinkedIn →