Blog / What Is Regression Testing? Types, Examples, and When to Run It
Guide

What Is Regression Testing? Types, Examples, and When to Run It

Regression testing guides list up to a dozen types and leave you to choose. Here are the three questions behind the names and the run to pick for each change.

You shorten the session timeout on Tuesday. By Thursday, support is forwarding screenshots of a checkout button that spins forever for anyone who has been signed in for an hour. Nobody touched checkout. The change was safe where it was made and wrong where it landed.

Regression testing is how you find that out on Tuesday. The skill that matters is deciding what to re-run on each change, and this guide turns the long list of types into that decision for a pull request and for a release.

What you’ll learn

  • What regression testing is and how it differs from retesting
  • The three questions behind the seven names for regression test types
  • What to re-run on a pull request, a merge, a nightly build and a release
  • What to automate, what to keep manual, and where autonomous testing fits

What Is Regression Testing?

Regression testing is re-running tests you already have after a change, to catch what the change broke in parts of the product it was never meant to touch. The ISTQB glossary describes it as change-related testing that looks for defects introduced or uncovered in unchanged areas.

A regression is the bug it hunts, meaning a feature that worked yesterday and doesn’t today. The new feature itself is not the target. Tests for the new feature ask whether the thing you built works, and regression tests ask whether everything else does too.

The term people confuse it with is retesting, which ISTQB also calls confirmation testing. The two run in sequence after a fix and answer different questions.

RetestingRegression Testing
QuestionIs this bug fixed?What else did the change break?
Tests runThe exact case that failedCases in areas the change should not have touched
SizeOne caseA chosen set, from a handful to a full suite

Run the retest first. There is no point checking the neighbors of a fix that didn’t work. Smoke testing, sanity testing and unit testing get mixed up with regression testing too, and each answers a narrower question.

CheckThe Question It AnswersWhere It Sits
Smoke testingIs this build stable enough to test at all?First, on every new build
Sanity testingDoes the area I just changed behave sensibly?A quick pass before deeper work
Unit testingDoes this piece of code do what it should?While writing, and on every commit
Regression testingWhat else did the change break?After a change, at the scope you choose

A Regression Test in Practice

Take the timeout as a worked example. The change lived in session handling and the failure showed up in checkout, so the test that catches it is neither a login test nor a payment test. It is a flow that crosses both.

  1. Sign in: Start a normal session.
  2. Age the session: Move the clock past the new timeout.
  3. Add an item: Put something in the cart.
  4. Pay: Complete checkout.
  5. Expect: The payment goes through, or the app asks the user to sign in again and keeps the cart. A spinner fails the test.

Someone writes that test once, after the Thursday screenshots. From then on it runs after any change that touches sessions or the cart, and the next person to shorten a timeout hears about it from a failed build instead of from support.

Why a Regression Costs More Than a New Bug

A bug in a new feature meets only the people who have tried it so far, and a regression meets everyone who already relied on the old behavior. The more often you ship, the more chances one has to get through, and the less time anyone has to check by hand.

Software is interconnected, so a change in one place has consequences in places its author never read. Martin Fowler’s writing on self-testing code argues that a test suite is valuable for the confidence it gives you to change the system. Confidence is what lets a team make a bigger change on a Friday. Without it, teams either slow down or ship and hope, and both cost more than the suite did.

Types of Regression Testing: Seven Names, Three Questions

Guides list anywhere from two types of regression testing to a dozen, and the lists disagree because the names answer three different questions. Sort the names by the question and the list stops being a list.

The QuestionNames You’ll SeeWhat They Mean
How much do you re-run?Retest-all, complete, selective (regional), partialRetest-all and complete run everything, with complete usually saved for a release or a core change. Selective, regional and partial run only the tests near the change.
Which tests do you run?Corrective, progressiveCorrective reuses the existing tests because the specification didn’t change. Progressive adds new tests because it did.
At what level?UnitUnit regression re-runs one unit’s tests with its dependencies stubbed, before the pieces are combined.

The level changes what a run costs and what it can see. Ask which level could have caught the checkout failure from the opening.

  • Unit: Fast enough to run on every commit, and blind to anything outside the one piece of code. It cannot see the checkout failure.
  • Integration: Checks that components still agree with each other, which is where a changed interface shows up.
  • System: Drives the product the way a user does. It is the slowest level per test and the level that sees a session timeout break a payment.

Two names on longer lists sort differently. Visual and non-functional regression change what a test checks, such as how a screen looks or how fast it responds, and not the scope, the test choice or the level. They run as separate suites alongside this one.

A retail company’s QA team described their setup to us. Their regression suite was split by module. Their smoke suite was a smaller version of it, and on Saturdays they kicked off an on-demand health check. No type from a glossary appears anywhere in that setup, only a scope for each trigger.

Teams don’t pick one type. They answer each question for each change, and a fourth question, who does the running, is the last section here.

Stop Hand-Running the Same Flows

Give Pie the flows you re-click by hand. It starts the run for you.

What to Re-run on Each Change

What you re-run on a given change depends on what the change can reach and how long anyone will wait to find out. Two techniques do the work.

  • Regression test selection: Runs only the tests a change can affect, found by mapping which tests exercise which code.
  • Test prioritization: Keeps the full set but reorders it so the likeliest failures come first, which shortens the wait for bad news and leaves the total runtime alone.

Google runs selection at a scale few teams need. In Software Engineering at Google, the chapter on continuous integration says its test system handles more than 50,000 unique changes a day and runs more than four billion test cases. It decides what to run by analyzing the downstream dependency graph of each change. Selection is only as safe as that map.

Suppose the sign-in-then-pay test from earlier is already in the suite when the next timeout change lands. The same change gets three different outcomes depending on the scope you chose.

Scope You ChoseWhat RunsCatches the Checkout Failure?
Selective, by changed filesTests mapped to the session codeOnly if the sign-in-then-pay test is mapped to that code
Selective, by critical flowSign-in, checkout and the other money pathsYes
CompleteThe entire suiteYes, after the suite’s full runtime

The scope usually lines up with the trigger like this. Treat it as a default, because a risky change earns a wider run than its trigger says.

TriggerScopeWhy
Pull requestSelective, the flows nearest the diffFast enough that the author hasn’t moved on
Merge to mainBroader, every critical-path flowCatches what two separate pull requests did to each other
NightlyThe entire suite, if it finishes overnightDuration matters less than having a result by morning
Release candidateThe entire suite, plus a person on anything a script can’t judgeThe last point where a regression is cheap to fix
Dependency, config or environment changeCorrective, the existing tests and no new onesNothing in your code changed, so the risk is the ground under it

The last row is easy to forget. When a test environment drifts away from production, a green run from last week tells you less than it did. On mobile the change that breaks things is often not yours at all. A new OS release or a different screen size is enough, so mobile regression testing has failure modes of its own.

Why Narrow Beats Complete on a Pull Request

Look, a narrow run you actually wait for beats a complete run you skip. Run the narrowest scope that could catch the change’s side effects on every pull request, and run the whole regression suite before a release.

The objection is that narrow is unsafe, and it has a point. A side effect outside your map gets through until the wider run at merge or release catches it. But a run nobody waits for is less safe, because skipping is a cost decision. A run that costs more minutes than the change looks risky gets bypassed, and every regression ships inside a change that looked safe.

A skipped run protects nothing.

A seatbelt is the right comparison. You don’t judge one by how many crashes it has stopped. It works because it gets worn on every drive, and it gets worn because buckling costs almost nothing. A run that comes back after the author has moved on gets skipped the first hurried Friday. So does a flaky test that teaches the team to read red as noise.

What a Green Run Does Not Prove

A passing run proves only that the checks you wrote passed. It says nothing about the checks you never wrote, and a suite can stay green while a screen is visibly broken, because every assertion looked somewhere else. A suite guards only what someone once thought to ask. The bugs outside that list belong to exploratory testing, which is a different discipline with a different payoff.

Where the tests come from is a separate job. Choosing the cases and structuring them is what building a regression test suite covers.

Manual, Automated, and Autonomous Regression Testing

Regression tests are the strongest candidates for automation, because the same checks run on every change and the expected result rarely moves. The real decision is who keeps the automation alive.

A person re-clicking the same paths by hand is slow, and it is the first thing dropped under deadline. What stays manual is whatever a script can’t judge, such as a feature in design or the call on whether a flow feels broken even though every assertion passed. If you’ve decided to automate regression testing, the five steps in that guide are the next read. The choice has three options, and they differ in who does the upkeep.

DimensionManualScripted AutomationAutonomous (Pie)
Who writes the testsA tester, as a checklistAn engineer, in codePie drafts them, from a plain-English description or its own discovery of the app
What a UI change doesA person adapts on the spotSteps bound to selectors break until someone fixes themSupported tests have no locators to break
CadenceOnce per release cycle, if at allEvery pull request, if the suite is fastOn a schedule or a new build, once configured
Best fitJudgment calls and new featuresStable UIs with engineers to spareFast-changing apps with no team to maintain scripts

Scripted automation has an upkeep problem built in. Whenever a UI change moves or renames an element, its selector breaks, and someone has to find which tests that broke before the suite means anything again. Self-healing test automation goes after that repair work, with tests meant to survive a UI change.

Which product to buy is a separate question. The ten regression testing tools we compared are sorted by the way suites fail rather than by feature count.

Let Pie Start Every Regression Run

The trigger table only protects you if something starts the run every time. Pie can own that step, so the scope you chose for each trigger runs without anyone remembering to run it.

  • Schedule it: Put the suite on a Loop, Pie’s saved set of suites and triggers, and pick a nightly or weekday time. The nightly row of the table then needs nobody to start it.
  • Start it from CI: A CI job on each merge starts the Loop with one request. For a mobile app, the job sends the build along with it.
  • Test the pull request: Comment @pie-pr-bot test on GitHub or /pie on GitLab to run Pie against the change. A web app needs a preview URL for the pull request, and a mobile app needs an uploaded build.
  • Fix what fails: Results land in Run History for each suite, and failures group into issues. Press Fix and Pie opens a Fix PR in a repository you connect, and merging stays your team’s call. Or send the issue to Jira, Linear or Asana.
CI pipeline firing a Pie Loop with one request, then a new run
One request from a CI job starts a Loop, and a new run begins.
Pie Run History showing pass, fail and issue counts for four test suites
Run History lists each suite with its passes, failures and issues.

Flows that need code-level control stay in your own framework, and the calls a script can’t judge stay with your testers.

Find the Regression on Tuesday, Not Thursday

Your regression suite is a promise that the parts of your product you didn’t touch still work. The promise breaks on cost. A suite that takes longer than anyone will wait gets skipped, and then the Thursday screenshot arrives.

Runtime is one cost and upkeep is the other. A suite nobody keeps up stops matching the product.

Pick the smallest run that could catch your next change. Run it every time. Run all of it before you ship.

Test the Flow That Breaks Most

Bring the flow you re-check every release. Watch Pie run it.

Frequently Asked Questions

Regression testing is re-running tests you already have after a change, to check that features which worked before keep working. It does not test the new feature itself. It looks for regressions, meaning something that worked yesterday and broke today because of a change somewhere else.
Retesting re-runs the one case that failed to confirm a specific bug is fixed. Regression testing re-runs a broader set across areas the fix should not have touched, to catch side effects. Retesting asks whether the bug is gone. Regression testing asks what else broke.
The seven common names are retest-all, complete, selective, partial, corrective, progressive and unit regression testing. They answer three questions about scope, test choice and level. Pair a selective run on each pull request with a complete run before release.
A team shortens a session timeout. Checkout, which nobody edited, starts failing for users who have been signed in for an hour. A test that ages a signed-in session past the new limit and then pays would have caught it before release.
After any change that could reach existing behavior, which includes dependency upgrades and environment changes as well as code. Run a selective set on every pull request and the complete suite before each release. Add a broader run on merge if two pull requests can collide.
Yes, and it is the strongest candidate for automation because it is repetitive and runs on every change. Keep a person on new features and on judgment calls a script cannot make. The hard part is upkeep, since scripts bound to selectors break when the UI changes.
Pie drafts tests from a plain-English description or from its own discovery of your app, so you are not scripting every case by hand. You group the flows you want into a suite and run it on a schedule, from CI or on demand.
Selenium and Cypress tie each step to a selector such as an XPath or a CSS path, so a UI change means someone fixes the locator. Pie re-reads the screen on every run, so a button that moves is not a locator someone has to repair.
Dhaval Shreyas
Dhaval Shreyas
CEO & Co-founder at Pie

13 years building mobile infrastructure at Square, Facebook, and Instacart. Now building the QA platform he wished existed the whole time. LinkedIn →