Blog / What Is Retesting? How to Confirm a Bug Fix Before You Run Regression
Guide

What Is Retesting? How to Confirm a Bug Fix Before You Run Regression

One green run does not prove a bug is fixed. See how deep to retest, how many clean runs an intermittent fix needs, and what to run once the retest passes.

Your developer moves a ticket to Resolved, you run the failing test once, and it passes. You close the ticket. Two weeks later the same bug is back in a customer’s screenshot, because it only ever failed one run in ten.

A passing run feels like proof. It only counts when the retest could have failed. Here is how to retest a fix so that a green run means the bug is gone.

What you’ll learn

  • A plain definition of retesting, and its other name, confirmation testing
  • Three depths of retest, and which one a given fix needs
  • The number of clean runs it takes to trust a fix for an intermittent bug
  • What to run after a passing retest, and how Pie keeps watching the bug

What Is Retesting?

Retesting is re-running the tests that failed because of a defect, on the build that contains the fix, to confirm the failure no longer happens. ISTQB calls it confirmation testing, and its Foundation Level syllabus (v4.0.1) says confirmation testing “confirms that an original defect has been successfully fixed.”

A retest has one narrow question and two possible answers. The bug is gone or it is not. The steps that trigger it are already known, because someone wrote them down when the test failed, and the retest depends on that record. A bug report with no reproduction steps cannot be retested, only guessed at.

Bugs found by exploratory testing show the point clearly. No scripted case failed, so the steps in the report become the test case. Write them as carefully as you would write a test.

Retesting vs Regression Testing

Retesting asks whether this bug is fixed. Regression testing asks what else the change broke. Both run after a change and both re-execute existing tests, which is why they blur in conversation. They differ in what triggers them, how big they are, and what a pass tells you.

DimensionRetestingRegression Testing
QuestionIs this one defect gone?Did the change break anything that worked?
TriggerA defect was marked fixedAny change that could reach existing behavior
Tests chosenThe ones that failed, or the reproduction stepsA set picked by impact analysis, from a handful to the full suite
Known in advanceNo, it depends on what failedYes, the suite exists before the change
A pass meansThe reported failure no longer occursNo adverse consequence showed up in what you ran
A pass does not meanThe fix caused no side effectsThe original bug is fixed

The order follows from the two questions. A regression pass around a fix that never worked only tells you the neighbors are fine while the bug is still there. The syllabus agrees, saying regression testing covers a change “including a fix that has already been confirmation tested.”

Retest First, Then Regress

Retesting confirms one fix, and regression testing checks everything around it. Run the retest first and let its result decide whether the regression run is worth starting.

How Much Should You Retest?

Retest to the depth the risk calls for. The ISTQB syllabus scales confirmation testing by risk, from the bare reproduction steps up to new tests for whatever the fix changed.

DepthWhat you runFits when
Reproduction stepsThe steps from the bug report, on the fixed buildThe fix is small and the release is close. The syllabus allows this fallback when time or money is short.
Every failed testAll tests that failed because of the defectOne defect failed several tests, so one green case says little about the rest.
Failed tests plus new onesThe above, plus new tests for what the fix addedThe fix introduced a rule or state that no test covered, such as a validation that did not exist before.

Two rules hold at every depth. Run against the build that holds the fix, and use the same steps and data that failed. Change the data and you have written a fresh test that happens to pass. The syllabus also prefers that the person who ran the original test runs the confirmation, since they know what the failure looked like.

How to Retest a Fix

A retest takes five moves, and the second one is the one people skip. Each move below takes minutes once the bug report is good.

  1. Find the fixed build: Match the build number in the ticket to the build you are testing. A retest on the previous build fails for the wrong reason.

  2. Watch it fail first: Run the reproduction on the build without the fix, or check that the original failure happened on the build just before the fix. A check that cannot fail cannot confirm anything.

  3. Same steps, same data: Keep the account, the inputs and the environment the report names. Add any missing setup step to the report before you run.

  4. Count the runs: One clean run settles a bug that failed every time. An intermittent bug needs more, and the next section shows how many.

  5. Close or reopen: On a pass, close the ticket and hand off to regression. On a fail, reopen it with the new run attached, so development sees what changed.

Stop Closing Tickets on One Run

Track the bug in Pie. Every later run checks the fix again.

How Many Passes Prove a Fix?

A bug that fails every time needs one clean run. A bug that fails sometimes needs arithmetic. A founder on one of our calls described a client’s app that flipped between broken and fine from one day to the next. A single retest of an app like that settles nothing.

Say a checkout error shows up in one run out of ten. With a fix that did nothing, each new run would pass nine times out of ten anyway, so a single green run looks the same with or without the fix.

Stack clean runs and the odds shift. The chance that an ineffective fix passes n runs in a row is (1 − p) raised to the power n, where p is how often the bug failed before the fix. Here p is 0.1, so each row below is 0.9 raised to the number of runs.

Clean runs in a rowChance an ineffective fix passes them all
190%
559%
1035%
2012%
304%

The shortcut generalizes. For a bug that failed one run in N, about 3N clean runs leave under a 5% chance of a false all-clear, whatever N is. It is the statistician’s rule of three: with no failures in n runs, the true failure rate could be as high as about 3 in n at 95% confidence. A one-in-ten bug needs 30 runs, and a one-in-twenty bug needs 60.

The 0.9 is an assumed failure rate, so swap in your own bug’s rate. It varies from case to case. The math also assumes each run fails independently, and real flakes often cluster around timing, data or a shared environment, so vary whatever the original failure varied. Intermittent failures can also come from the test itself, and the guide to flaky tests covers that side, while how to fix flaky tests covers the repair. When you cannot afford 3N runs, write the count on the ticket (“passed 5 of 5, not confirmed”) and leave it open for monitoring.

What Runs After a Retest Passes

A passing retest earns the regression run. An impact analysis picks which cases from your regression test suite to run, scoped to what the fix touched:

  • Changed area: The code the fix edited, and the screens that show it.
  • Neighboring flows: The steps a user takes just before and after the fixed one.
  • Shared code: Any other path that calls what the fix changed.

The reproduction should join that suite too. Once a bug has shown up, it has proven it can happen, so the retest becomes a permanent check. This is also where the automation question gets its answer. A retest cannot be scripted before the failure exists. The day after, it can, and how to automate regression testing covers the steps.

Let Pie Re-Check Every Tracked Bug

Tracking a bug in our agentic testing platform turns the retest into something that keeps running. Pie builds a repro test for the issue, and your later runs carry it forward.

  • Track the bug: On a real bug, select Track it. Pie builds a repro test, a test case that reproduces the bug.
  • Watch the repro: Pie re-checks each tracked issue on every run. While the repro test fails, the issue shows Repro and the bug is alive.
  • Read the verdict: Once the repro no longer finds the bug in the most recent run, the issue shows Looks resolved. You pick Mark as resolved or Keep tracking.
  • Ask from your agent: The MCP tool get_issue_repro_history returns a verdict, its evidence source and the recent trend, so a coding agent can check the fix it just wrote.
Pie issue, agent fix, pull request, repro check, then likely resolved or still reproducing
Once the fix lands, Pie’s repro check says likely resolved or still reproducing.

When I walk a team through a tracked issue that has stopped reproducing, the answer goes one of two ways. Their team fixed it, or it is an intermittent bug that skipped this run and belongs on the list a while longer. Keep tracking is for the second case, and each later clean run adds to the 3N count.

Close the Ticket on Evidence

Retesting is the smallest check in testing, and the easiest to close too early. Run the same steps on the build that holds the fix and match the depth to the risk. When the bug fails only some of the time, count the passes.

The count is the part that slips. Thirty runs for one checkout bug rarely fit in a sprint, so the ticket closes on five and the bug comes back in a customer’s screenshot.

A tracked issue in Pie has its repro checked on every run, so the count builds from runs your team was doing anyway. How many of the tickets your team closed this month passed exactly once?

Know When a Fix Holds

Close tickets on a count. Pie keeps the repro running.

Frequently Asked Questions

Retesting is re-running the tests that failed because of a defect, on the build that holds the fix. Say checkout failed for a saved card. Once the fix ships, you repeat those checkout steps with the same card on the new build. A pass means that one defect is gone.
Yes. Confirmation testing is the name the ISTQB Foundation Level syllabus uses for the same activity. Both describe re-running the failed tests against the fixed build to confirm that the defect is gone and the failure does not occur again.
Retesting re-runs the failed tests to confirm one reported defect is fixed. Regression testing checks that the change did not break existing behavior, using a set chosen by impact. A passing retest says nothing about side effects, and a passing regression run does not prove the bug is gone.
Retest first. A failed fix goes back to development and saves you the regression run. A fix that works earns a regression pass around the changed area. The ISTQB syllabus describes regression testing as covering a fix that has already been confirmation tested.
Enough runs that a still-broken fix would be unlikely to pass them all. For a bug that failed one run in N, about 3N clean runs in a row leave under a 5% chance of a false all-clear, assuming independent runs. A one-in-ten bug needs 30 runs.
Yes. Reusing the original steps, data and environment ties the result to the defect. Change the data and a pass could mean the fix worked, or that you simply avoided the failing path. Add new tests alongside it when the fix changed behavior.
Automation is possible once the bug exists. You cannot script a retest before the failure, but you can script the reproduction afterward and keep it in your regression suite so the bug cannot return unnoticed. Until then, the steps from the bug report are the retest.
When you track an issue, Pie builds a repro test and re-checks it on every run. Once the repro no longer finds the bug in the latest run, the issue shows Looks resolved. You then choose Mark as resolved or Keep tracking, so your team makes the final call.
Jinoo Jain
Jinoo Jain
CPO & Co-founder at Pie

Spent a decade in B2B SaaS sales before building Pie. Now obsessed with helping engineering teams ship without the fear of breaking things. LinkedIn →