Blog / Regression Test Selection: How to Run Fewer Tests Without Missing Failures
Guide

Regression Test Selection: How to Run Fewer Tests Without Missing Failures

Regression test selection always has a change it cannot see. Pick tests per change and audit your selector against a full run so it misses no failures.

A test selector has one job: run only the tests a change can break. We wrote one for a small shop app, 32 lines on top of coverage.py, and changed one line in a config file. It picked 1 test out of 12, and that test passed. A full run of the same commit failed 4. The selector had no test mapped to that file.

Every way of picking regression tests has a change it cannot see. Here is how to pick tests per change and prove your selector is not hiding the failures that matter.

What you’ll learn

  • The five signals that decide which regression tests run for a change
  • A rule for which kinds of change need every test
  • Three real misses from a selector script we wrote and ran
  • A 17-line audit that catches a selector skipping failures

What Is Regression Test Selection?

Regression test selection is choosing which tests from an existing suite to run for one specific change, based on what that change can affect. It cuts run time, and it adds one risk. A test that does not run cannot fail, so if the selector leaves out a test that would have failed, the bug gets through.

Two other practices sound similar. Prioritization runs every test and puts the likely failures first. Minimization deletes redundant tests from the suite for good. Selection keeps the whole regression suite and, for each change, runs only the slice that change can affect. Because the full suite is still there, you can audit a selector against a full run.

The pressure to select shows up once the suite is the slowest step in the pipeline, the pattern behind append-only suites. Selection starts the moment a change lands and someone has to decide which regression tests run.

The short version

Run the union of five signals, force a full run for any change the signals cannot see, and compare against a full run to catch every miss.

Five Signals for Picking Tests for a Change

Five signals decide what a selector picks. Each answers a different question about the change, and each has a blind spot.

SignalThe question it answersWhere it skips a real failure
Changed filesWhich tests live next to the files in the diff?Shared code, and any behavior a test reaches through another module
Coverage or dependency mapWhich tests execute the code that changed?Config, data files, dependency upgrades, and anything built after the map was
RiskWhich flows cost the most if this change broke them?A judgment call that goes stale and ignores what the diff touched
Failure historyWhich tests failed lately and could fail again?New code with no history, and flaky tests that look like signal
Always-run smokeWhat must work after every change?Depth: a passing smoke test says nothing about the flows below it

Changed Files

The cheapest signal is the diff itself. A rule that maps src/checkout/** to tests/checkout takes minutes to write and breaks the first time a change lands in shared code. Playwright’s --only-changed flag goes one step further. Its release notes say it runs the test files changed since a git ref, plus every test file that imports a changed file. It follows imports, not what a test executes, so a change that reaches a test any other way stays invisible to it.

Coverage and Dependency Maps

A map records which tests execute which files. It works almost like a book’s index, where you look up a file and find every test that ran it. Datadog’s Test Impact Analysis builds it from per-test coverage and cross-references it with the files in a new commit. Google’s CI chapter describes the same idea at a larger scale, where the main mechanism is analysis of the downstream dependency graph.

A map only knows what tests executed. Datadog’s docs list what its default setup cannot detect: library dependencies, compiler options, external services, and data files used by data-driven tests.

Risk

Risk asks what it would cost if this area broke. Authentication, checkout, and payments earn a place in every selection, whatever the diff says. A risk list is one person’s judgment, so give it an owner and a review date, or it describes last year’s product.

Failure History

Tests that failed recently are good bets to fail again, so they ride along with the next change. Azure’s Test Impact Analysis includes previously failing tests next to the impacted and newly added ones. Our selector reads a plain list of recent failures and adds them to the pick. Quarantine flaky tests first, because history carries them along too.

Always-Run Smoke

A small tagged tier runs on every change, whatever the diff says. Treat it as the floor under the other signals, the same job a smoke test does anywhere. Its limit is depth. In our run the smoke test passed while four other tests failed.

A Rule for Each Kind of Change

Pick the union of the signals, then ask whether the map can see this change. When it cannot, run everything.

The changeWhat runsWhy
A source file the map coversTests that cover it, the smoke tier, recent failuresThe map has seen this file run
A new or edited testThat test, plus the pick aboveA new test has no coverage or history yet
Config, data files, flags, CI settingsEverythingCoverage never measured those files
A dependency or lockfile upgradeEverythingDatadog lists library dependencies as invisible by default
A new module, or code moved between filesEverything, then rebuild the mapThe map has never seen the new path
Markup or styles onlyEverythingCoverage never ran markup or styles, and Azure’s analysis falls back to all tests on HTML and CSS changes

What a Selector Misses: A Small Project We Ran

We built a small shop app with 12 pytest tests and a tax rate in a JSON file, then wrote a 32-line selector on top of coverage.py’s per-test contexts. The script takes the files in git diff, looks up the tests that ran lines of them, and adds the smoke tier and a list of recent failures. A change to a tracked file runs every test.

"""Pick tests for a change: smoke, tests that cover the changed files, recent failures."""
import fnmatch
import subprocess
import sys
from pathlib import Path

from coverage import CoverageData

base, tracked = sys.argv[1], sys.argv[2:]
changed = subprocess.run(["git", "diff", "--name-only", base],
                         capture_output=True, text=True, check=True).stdout.split()

# Files the map cannot see (config, lockfiles): any change to them runs everything.
if any(fnmatch.fnmatch(f, g) for f in changed for g in tracked):
    print("tests")  # the whole suite
    sys.exit()

data = CoverageData(".coverage")
data.read()
covers = {}  # file -> tests that executed a line of it
for path in data.measured_files():
    rel = Path(path).resolve().relative_to(Path.cwd()).as_posix()
    for contexts in data.contexts_by_lineno(path).values():
        covers.setdefault(rel, set()).update(c.split("|")[0] for c in contexts if c)

smoke = subprocess.run([sys.executable, "-m", "pytest", "--collect-only", "-q", "-m", "smoke"],
                       capture_output=True, text=True).stdout.split()
history = Path("tools/recent-failures.txt")
recent = history.read_text().split() if history.exists() else []

picked = {t for f in changed for t in covers.get(f, ())} | {t for t in smoke if "::" in t} | set(recent)
print("\n".join(sorted(picked)))

Build the map once, then pick for each change and run what it picked:

COVERAGE_CORE=ctrace python -m pytest --cov=shop --cov-context=test
python tools/pick_tests.py HEAD "config/*" > picked.txt
python -m pytest @picked.txt --junitxml=selected.xml
python -m pytest --junitxml=full.xml

We ran it on Python 3.14.0 with pytest 9.2.0, pytest-cov 7.1.0, and coverage.py 7.17.2. Each row below is a change we made, the pick, and what the full run of the same commit said.

The changeTests pickedFailed in full runFailed in selected run
Coupon rate in discounts.py from 10 to 154 of 1211
US tax rate in config/tax.json, no tracked files1 of 1240
Same change, config/* tracked12 of 1244
rounding.py switched from rounding to flooring, map built before the refactor1 of 1210
Same change, map rebuilt after the refactor4 of 1211

A Config File Nobody Mapped

The tax rate lives in config/tax.json. Coverage measures Python files, so the map has no entry for it, and the diff matched nothing. The selector fell back to the smoke test, which passed. The full run failed four tests, from the pricing unit test up to the checkout totals. Listing config/* as tracked makes the same change run all 12, which is the fix Datadog’s docs call tracked files.

A Map Built Before the Code Moved

We refactored rounding out of checkout into a new shop/rounding.py and did not rebuild the map. A later change to that file matched nothing, because the old map had never seen it. Only the smoke test ran, and the full run failed the tax-rounding test. After we rebuilt the map on the refactored commit, the same change picked 4 tests and the rounding test failed where it should.

A Map That Was Quietly Incomplete

Our first map build on Python 3.14 used coverage.py’s default sys.monitoring core. It raised a single warning in the pytest summary, and the pick on the coupon change had 2 tests where the right answer was 4, so the audit flagged the coupon test as missed. coverage.py’s own message reference says dynamic contexts are not supported with that core and the context data may be incomplete. Setting COVERAGE_CORE=ctrace fixed it. The build exited cleanly, so read the map builder’s warnings as part of the selector.

Audit a Suite You Never Scripted

Point Pie at your app. Get end-to-end tests worth auditing.

How to Check a Selector Is Not Skipping Failures

All three misses surfaced the same way, as full-run failures in tests the selected run never ran. Comparing the two runs is the entire audit. Microsoft’s Azure Pipelines docs describe the same check for impact analysis: run the selected tests, run all tests, and confirm the full run reports the same failures.

"""Compare the failures of a selected run with a full run of the same commit."""
import sys
import xml.etree.ElementTree as ET


def failed(path):
    return {f"{c.get('classname')}.{c.get('name')}"
            for c in ET.parse(path).getroot().iter("testcase")
            if c.find("failure") is not None or c.find("error") is not None}


full, picked = failed(sys.argv[1]), failed(sys.argv[2])
missed = sorted(full - picked)
print(f"full run failed {len(full)}, selected run failed {len(picked)}, missed {len(missed)}")
for name in missed:
    print("  MISSED", name)
sys.exit(1 if missed else 0)

On the config change with no tracked files, python tools/audit_selection.py full.xml selected.xml printed this and exited with code 1:

full run failed 4, selected run failed 0, missed 4
  MISSED tests.test_checkout.test_checkout_rounds_tax_to_nearest_cent
  MISSED tests.test_checkout.test_checkout_us_total
  MISSED tests.test_checkout.test_checkout_with_coupon
  MISSED tests.test_pricing.test_tax_rate_known_region

Turn it into a routine in five steps:

  1. Shadow the selected run with a full run. Run every test on the same commit at merge or nightly, whichever you can afford. Azure’s docs recommend a full run at a set interval, and Datadog leaves selection off the default branch so every test runs before production.
  2. Diff the failures. A miss is any test that failed in the full run and did not fail in the selected run.
  3. Name the cause of every miss. A file type the map cannot see goes on the tracked list. A stale map gets rebuilt. An incomplete map sends you back to the builder’s warnings.
  4. Rebuild the map from the full run. The nightly run with coverage contexts feeds the audit and the map, so the second job rides on a run you already do.
  5. Count misses per week. The target is zero. A clean week with few changes proves little, because the audit only sees failures that changes caused, so judge the selector over many changes.

Pie Runs the End-to-End Side of the Check

Pie adds an end-to-end layer that works alongside your QA team and your unit tests. Our agentic testing platform explores your app and writes tests for web, iOS, and Android, so the suite you audit does not wait on someone to script it.

  • Test the change itself: Comment @pie-pr-bot test on a GitHub pull request or /pie on GitLab, and Pie tests the change. It needs something to run against: a preview URL for a web app, or an uploaded build for a mobile app.
  • Run the end-to-end suite on a schedule: A Loop runs the suites you pick at the time you set, such as every night. That nightly run is the full-suite baseline for your end-to-end layer.

A Selector Is a Bet You Can Audit

Picking tests per change pays for itself, and every method is a bet on what a change can touch. Coverage maps lose to config files, path rules lose to shared code, and risk lists lose to last quarter’s product.

Shadowing each pick with a full run, and counting what the selector missed, keeps the bet honest. Ours missed tests three ways, and each miss named its own fix.

A skipped test is a promise that it would have passed. Check the promise.

Run the End-to-End Suite Nightly

Put the end-to-end run on a Loop. Keep your selector for unit tests.

Frequently Asked Questions

Regression test selection is choosing which tests from an existing suite to run for one specific change, based on what that change can affect. It cuts run time, and it adds one risk. If the selector leaves out a test that would have failed, the bug gets through.
Run the union of five signals. A coverage map picks the tests that executed the changed code, and risk, recent failures and a small smoke tier add the flows a diff cannot point to. Force a full run for changes no signal can see, such as config files.
Selection chooses which tests run for one change. Prioritization keeps every test and reorders them so likely failures come first. Minimization removes redundant tests from the suite for good. Selection and prioritization leave the suite intact, which lets you audit a selector against a full run.
Test impact analysis is a coverage-based form of selection. It records which code each test executes, compares that with the files in a commit, and runs only the affected tests. Datadog and Azure Pipelines both ship it, and both document changes it cannot see.
Yes. A selector skips a test that would have failed whenever a change sits outside its map. In our small project, a config file edit failed four tests while the selector ran only the smoke test. A full run on the same commit exposes the miss.
Run the selected tests, then the full suite on the same commit, and compare failures. Any test that failed in the full run and not in the selected run is a miss. Name its cause, fix the map or the tracked-file list, and aim for zero misses.
Changes the map cannot see, including config and data files, dependency or lockfile upgrades, CI settings, and code moved into new files. Datadog lists library dependencies and data files as invisible to its default setup, and Azure falls back to all tests on HTML and CSS changes.
Yes, for the end-to-end suites you build in Pie. A Loop runs the suites you pick at the time you set, such as every night, so a full run is always there to audit against. Your unit and integration suites stay in your pipeline with their own selector.
Adithya Aggarwal
Adithya Aggarwal
CTO & Co-founder at Pie

Eight years building search and delivery systems at Amazon. The kind of scale where flaky tests block billion-dollar releases. Now CTO at Pie, building AI agents that adapt when your UI changes. LinkedIn →