Blog / Regression Testing in CI/CD: What Runs on Every PR and What Waits Until Tonight
Guide

Regression Testing in CI/CD: What Runs on Every PR and What Waits Until Tonight

A green pull request and a red nightly run mean checks sit in the wrong lane. Give each lane a time budget and an owner so breaks surface at the PR.

Your pull request merged green on Tuesday. On Wednesday morning the nightly regression run is red. Four merges landed in between, and the morning goes to reading diffs to find whose change broke it.

The pull request run was too thin to catch the break, and the nightly run sat too far from the change to blame anyone. Here is how to give each lane a time budget and an owner, so the next red run points at one change.

What you’ll learn

  • The three regression lanes in a pipeline and what each one runs
  • How to set a budget and name an owner for the pull request, merge, and nightly lanes
  • How a nightly failure becomes a standing pull request check, with a worked example
  • Where lanes break, from shared test accounts to a silent schedule, and what a failure report should say
  • How to run the nightly lane in a Pie Loop

How Should Regression Testing Run in CI/CD?

Run regression testing in three lanes, plus a short pre-release check. A pull request lane runs a small, fast subset and blocks the merge. A merge lane runs the broader suite against the merged code. A nightly lane runs everything, including the slow and wide checks, and reports to a named owner. The pre-release check then runs the critical journeys on the release build before it ships.

Each lane answers a different question:

  • Pull request lane: Did this change break anything obvious?
  • Merge lane: Does the change hold up next to everyone else’s? That check is the core of continuous testing.
  • Nightly lane: Does the product work end to end, even on a day nobody touched it?

What Runs in Each Lane

Set the budget for each lane before you fill it. Developers wait on the pull request lane, so it has to finish while they wait. Martin Fowler’s Continuous Integration article cites a ten minute build as the XP guideline. The nightly lane has no one waiting, so it can carry whatever the other lanes cannot afford.

LaneTriggerWhat runsBlocks?Failure owner
Pull requestEvery push to the PRTests tied to the change, plus critical-path smokeThe mergeThe PR author
MergeMerge to mainThe broader regression suite on merged codeThe release candidateThe merge author, with on-call as backup
NightlyA scheduleThe full suite, plus slow and wide checksNothing, it opens a ticketA named owner for the lane
Pre-release checkA release candidateCritical journeys on the release buildThe releaseThe release owner

What belongs in the pull request lane is a judgment call, and it should stay short. Start with the tests that cover the changed code and the journeys that would hurt most if they broke, such as sign-in and checkout. Everything that duplicates those checks moves to a later lane. The workflow file is the easy part, and step 4 of how to automate regression testing shows one that shards the pull request run across four jobs.

The pre-release check is the only run that tests the artifact you ship. It runs the critical journeys on the release build itself, so a packaging or signing problem shows up before customers see it.

How to Set a Budget for Each Lane

A budget is a number the lane may not exceed, and it only holds if it comes from measurement. Record each lane’s run time for two weeks, then set the budget from the slow runs instead of the average. Developers remember the day the pull request lane took twenty minutes, not the nine days it took six.

LaneThe budget is set byFirst lever when over
Pull requestHow long a developer waits before switching tasksMove checks that duplicate each other to a later lane
MergeThe gap between merges, so runs do not queueSplit the suite across parallel runners
NightlyThe hour the team starts workAdd runners or split the suite across two scheduled jobs
Pre-release checkThe release windowTrim the journeys to the ones that gate this release

When a lane runs over, buy time in this order. Remove duplicated checks first, then split the suite across parallel runners, and cut coverage last. A budget also tells you how many tests fit, and the regression suite guide shows how to size a tier from its time budget and runner count.

Runners stop helping once the test environment becomes the limit. On one customer’s app, their own bot protection set the length of a regression run. Once heavier traffic got blocked, we had to run the suite a few tests at a time, and more runners would not have made it shorter. Budget the lane against what the environment accepts, not against the runners you can buy.

How a Nightly Failure Becomes a Pull Request Check

The lanes improve each other when failures move between them. A red nightly run is a pull request check that did not exist yet.

  1. Find the change: List the merges between the last green nightly and the red one, then narrow the range until one change fails.
  2. Write a small reproducer: A test that fails on that change alone is cheaper to run than the nightly test that found it.
  3. Add it to the pull request lane: Note why it is there, so the next person who prunes the lane knows which break it guards.

The same loop runs in reverse. When the pull request lane drifts over its budget, move the tests that duplicate each other to the nightly lane, and the gate stays inside its time. Pruning the suite keeps every lane honest.

Example (illustrative): Four merges land on Tuesday. At 02:00 the nightly run fails one test, “apply a coupon at checkout,” which was green the night before. The range is four merges, so the lane owner reruns the failing test on the merge commits, starting in the middle. Git’s bisect command can run that search automatically. The test passes on the second merge and fails on the third, which changed the rounding helper the coupon code calls.

The owner writes a small test that applies a coupon to a price with a rounding edge and confirms it fails on the third merge. It joins the pull request lane with a note naming the checkout break it guards. The next change that touches the helper meets that test before it merges, so the nightly run never has to find the same break twice.

Give the Nightly Lane an Owner

Schedule the full suite in a Pie Loop. Keep the gate small.

Five Ways Regression Lanes Break

Five problems break lanes, and four of them live outside the tests themselves:

  • Shared test accounts: Pull request and scheduled jobs that share logins or quotas collide, so a big nightly run can fail a late pull request. Give each lane its own accounts, or run them one after another.
  • Wrong blame: In GitHub Actions, scheduled workflows run on the default branch, so a red nightly points at a range of merges, not at the newest pull request.
  • Flaky tests: One flaky test in the blocking lane teaches everyone to rerun instead of read. Detecting and quarantining flaky tests is its own discipline.
  • Stale runs: A new push makes the old run meaningless. A concurrency group with cancel-in-progress lets the newer run cancel the older one, so the lane spends its budget on current code.
  • A silent schedule: GitHub disables scheduled workflows in a public repository after 60 days without repository activity. A nightly lane can go dark with nothing red to show for it, so check that it ran.

GitHub’s pull request event runs its workflows on the pull request’s merge branch. The lane then tests the code that would land on main, which is the code the gate exists to protect.

Match the Symptom to the Move

Each failure pattern points at a lane problem, and each has a first move.

SymptomLikely causeFirst move
The pull request lane keeps growingChecks that duplicate each otherMove the duplicates to the nightly lane
Pull requests pass, then the nightly run failsThe gate is too thin for what changesPromote the failing check into the pull request lane
The nightly run stays red for daysNo owner, or no way to find the causeName an owner and narrow the merge range
Developers rerun red builds without reading themA flaky test in the blocking laneQuarantine it to the nightly lane until it is fixed
No nightly result for a few daysA disabled or failing scheduleAlert when the run has not reported by a set hour

Wire the alerts narrow before you wire them wide. Start with the breaks that hurt, such as a nightly run that never reported or a checkout journey gone red. An alert on every failing test turns into noise the team learns to ignore.

What a Lane Failure Report Should Say

The nightly lane works almost like a night shift. Nobody watches it run, so the handover it leaves decides how fast the failure closes. A report that says only “nightly failed” sends the owner digging. A report with these fields starts them at the fix:

  • The test and the lane: Name the failing check and where it ran.
  • Last green and first red: Link both runs so the owner can compare them.
  • The merge range: List the merges between those two runs.
  • The suspect change: Add it when a bisect has already narrowed the range.
  • The evidence: Attach the log, screenshot, or trace from the failing run.
  • The owner and the next step: Name who has it and what happens next.

Run the Nightly Lane in a Pie Loop

Nobody waits on the nightly lane, which makes it the easiest one to hand off. Pie’s agentic testing platform explores your app and generates the tests, and a Pie Loop runs the saved suites you pick on a schedule, such as daily or weekdays.

  • A schedule per suite: Run the full regression suite nightly and give a short critical-path suite its own schedule.
  • No scripts to write: Pie’s tests come with no locators to break.
  • New mobile builds: For iOS and Android apps, existing tests can rerun against a new build when the Loop’s new-build trigger is set.
Pie Loop card with a daily schedule, last run and next run
A Loop card shows its schedule, last run and next run.

Let the Lanes Teach Each Other

Regression testing in CI/CD works when each lane has a budget and an owner. The pull request lane stays small because the nightly lane catches what it misses, and every miss moves one check to the gate.

The nightly lane is the one that goes unowned, like the Wednesday run with four merges to read. A Pie Loop keeps that suite on its schedule, so whoever owns the lane starts the day from a finished run.

Over a quarter, the gate gets smarter without getting slower.

Keep the Gate Fast and the Nightly Honest

Hand the nightly run to Pie. Spend the gate's minutes on the change.

Frequently Asked Questions

It is rerunning existing tests automatically as code changes, to confirm that nothing which worked before has broken.

A fast subset runs on each pull request and the full suite runs on a schedule. A break then surfaces close to the change that caused it.

Usually not. Run the tests that cover the changed code plus a short set of critical journeys, such as sign-in and checkout.

The pull request lane has to finish while the developer waits, so the slow and wide checks wait for the merge or nightly lane.

The full suite, including the slow end-to-end flows and the wide browser or device sweeps.

A nightly run tests the main branch rather than one pull request, so it catches drift and interactions between merges that a small gate cannot see.

The pull request lane should finish while developers wait. Martin Fowler's Continuous Integration article cites a ten minute build as the XP guideline.

Nightly and pre-release runs can take longer because nobody is waiting on them. Set each budget from your slow runs, not the average.

No. A test that fails at random teaches developers to rerun instead of read. Quarantine it out of the blocking lane and keep running it nightly. Fix the root cause before it returns to the gate.

It can, with extra setup. Mobile lanes need a build to test, so the pipeline must produce one and hand it over.

Device or simulator capacity limits how much runs in the pull request lane, so keep the broad device sweep in the nightly lane.

Start by moving checks that duplicate each other to a later lane. Then split the suite across parallel runners, and cut coverage only as a last step.

Parallel runners stop helping once the test environment limits how much traffic it accepts at once.

Yes. A Pie Loop runs the saved suites you choose on a schedule, such as daily or weekdays, and a short critical-path suite can keep its own schedule.

For iOS and Android apps, existing tests can also rerun against a new build when the Loop's new-build trigger is set.

Adithya Aggarwal
Adithya Aggarwal
CTO & Co-founder at Pie

Eight years building search and delivery systems at Amazon. The kind of scale where flaky tests block billion-dollar releases. Now CTO at Pie, building AI agents that adapt when your UI changes. LinkedIn →