Blog / How to Test Vibe-Coded Software Nobody Has Read
How-To

How to Test Vibe-Coded Software Nobody Has Read

Vibe-coded apps ship code nobody has read, so tests that depend on knowing the code go soft. Here's what still catches bugs, and when an app is ready for users.

When Andrej Karpathy named vibe coding in a February 2025 post, he called it “not too bad for throwaway weekend projects.” A few lines earlier he’d written, “I don’t read the diffs anymore.” The habit outgrew the weekend. Apps built on Lovable alone now draw more than 900 million visits a month, by the company’s August 2026 count.

The bad news first. Most of your testing habits assume someone on the team understands the code, and vibe coding removes that person on purpose. So you test the running app from the outside, against what it’s supposed to do for the people using it.

What you’ll learn

  • Why vibe-coded software breaks the tests that assume someone read the code
  • Which tests still work, and where their definition of correct has to come from
  • A five-step test plan you can run without reading the code
  • What a vibe-coded app should prove before it takes real users, data or money

What Makes Vibe-Coded Software Different to Test

Nobody on the team has read it, and that matters more than the fact that an AI wrote it. The OpenSSF glossary defines vibe coding as accepting AI-generated code “without reviewing it or understanding it,” and says code a human has reviewed isn’t vibe coding. Karpathy admitted as much in his original post, writing, “The code grows beyond my usual comprehension.”

Hand-written code came with an owner of its meaning, the person who typed it. Writing it forced them to know what each piece was for, and our testing habits were built on that free byproduct.

Unit tests encode the author’s intent. Reviewers read the diff. Coverage targets assume someone chose what was worth covering. Vibe-coded software ships without an owner of its meaning at all.

QuestionAI-assisted, reviewedVibe coded
Who can explain how a module works?The engineer who reviewed itNobody on the team
Where does “correct” live?In the reviewer’s head, the tests and the pull requestIn the prompt history, if anywhere
What can testing lean on?Knowledge of the codeOnly what the app does when you use it

Many of the people building it aren’t engineers. Four in five of the Lovable users the company surveyed in May 2026 describe their roles as non-technical. Most developers Stack Overflow surveyed in 2025 kept it out of their day jobs, with 72% saying vibe coding isn’t part of their professional work.

Some engineers do it too, and we’re no exception. Earlier this year I pushed a change with Claude Code, and Adithya, our CTO, reviewed it with Claude Code. I answered his review with Claude Code too, and we shipped it. We don’t sit and read the code line by line.

When an engineer reads and understands what the AI wrote, you’re testing AI-generated code, and the usual review-plus-tests routine has a reader to lean on. Take that reader away and each of your checks needs another source for what correct means. Find that source before you write a single test.

Which Tests Still Work When Nobody Understands the Code

The tests that hold up get their definition of correct from outside the code. The ones that fail need someone who understands it, and on a vibe-coded app that person doesn’t exist. Here’s how the usual checks fare.

CheckWhat it needs someone to knowHolds up on vibe-coded software?
Code reviewWhat the change is supposed to doNo. There’s no reader, or the reader is another model
Unit tests written from the codeWhat each function should returnWeak. They encode a guess about intent
Tests the same AI generates from the codeNothing beyond the code itselfWeak. They tend to confirm what the code does
Coverage targetsWhich paths matterMisleading. Code that ran isn’t code that’s right
Static analysis and security scannersKnown bad patternsYes, for their own job. They can’t tell you a feature works
Tests written from the requirement, before generationWhat the user needsYes. A person supplied the intent
End-to-end checks on the running appWhat a user should be able to doYes. They never look inside

Code review drops out first, and reviewing the diff harder can’t help when the diff goes unread. Tests-first developers and end-to-end testers are on the same side here. Both put a person’s definition of correct somewhere the generated code can’t rewrite it. What fails is any test that learned what correct means by reading the code.

Researchers have measured that failure. In a study accepted at ISSTA 2026, a software testing research conference, handing a model buggy code to test made it write more tests that assert the bug and fewer tests that catch it. Swapping the code in the prompt for a model-written description of what it should do reduced the effect. In a small run of our own, we handed one function with one boundary bug to three free models through bare APIs, and every suite they wrote defended the bug.

When the Builder Tests Its Own Work

Some app builders now test their own output. Replit’s docs say App Testing lets its Agent “test the apps it builds using an actual browser,” and Lovable’s agent can run browser tests on your app too. Both beat no testing at all.

Either way, the product that wrote the app also decides what working looks like. Keep those tests, and add a check that doesn’t share the builder’s idea of correct.

What the Standard Testing Advice Costs

Look, the standard advice for a vibe-coded app is to write good integration tests, or to have the AI write them. The first costs exactly the understanding vibe coding skipped, since specifying how the parts should talk to each other means knowing the parts. The second costs nothing and adds little, because the model reads the same code to decide what correct means.

A test that learns from the code learns its mistakes too. It’s the difference between AI that writes tests and a check that uses the product. Sort your checks by where their idea of correct came from. Stop counting any that learned it from the code.

How to Test Vibe-Coded Software in 5 Steps

Exercise the app the way its users will, starting with the flows that cost you most when they break. Think of it as a test drive. You can’t open the hood and explain the engine, so you drive the car hard on bad roads, and again after every visit to the shop.

The bugs you find this way are rarely exotic. Last year Adithya had Lovable generate an office-furniture marketplace and let Pie run discovery on it. Clicking a product didn’t open its detail page, and one control wasn’t tappable. The form for listing an item took any value typed into it.

He fed the findings back into Lovable, and by his estimate it fixed every one within five minutes. The builder could repair what it hadn’t caught.

Five checks, in this order:

  1. Write down what correct means: List what a user must be able to do and what must never happen, in plain sentences, before you open the app. The prompt history isn’t a spec, and every later check grades the app against this list.
  2. Run the money and data flows end to end: Go through sign-up, login, payment and anything a user would hate to lose, on the deployed build in a fresh browser session. Use realistic data in place of the demo’s tidy rows, and check the outcome as well as the clicks. The order exists, the email arrives and the record survives a reload.
  3. Test as two users and as nobody: Copy the link to something private, then open it as a second user and in a logged-out window. Change an ID in a URL and see what loads. In late January 2026, Wiz found that Moltbook, a social network for AI agents whose founder said he didn’t write a line of its code, let any visitor read and change its database.
  4. Break it on purpose: Type a negative quantity, paste a very long name, double-click Pay, refresh halfway through checkout and hit Back after submitting. When security firm Tenzai had five coding agents build the same apps in December 2025, four of them let a shop order go through with a negative total, because the prompt never said quantities had to be positive.
  5. Rerun all of it after every prompt session: Coding agents change things you didn’t ask them to change. In SWE-CI, a 2026 benchmark preprint that replays months of real commit history, most models broke a test that used to pass in more than three of every four tasks. Keep a known-good version you can roll back to.

Pair the five with a scanner track that looks inside the code for known bad patterns. Run a dependency audit and a secret scanner over your front-end bundle, then review your database access rules, which is where Moltbook went wrong. Security vendor Escape scanned 5,600 public apps built on vibe-coding platforms in 2025 and found more than 2,000 vulnerabilities. It calls that a lower bound.

None of those scanners will tell you checkout works. If you only have an afternoon, start with steps 1 to 3.

When Is a Vibe-Coded App Ready for Production?

It’s ready when the five checks pass on the build you’re about to ship, and they’re set up to run again on the next one. Vibe-coded doesn’t mean unshippable.

What changes at production is who pays when something breaks. A weekend project fails in front of you. A production app fails in front of users, with their data and their money.

Karpathy drew the same line a year after coining the term. In a February 2026 post he wrote that when he named it, models were weak enough that “you’d mostly use vibe coding for fun throwaway projects, demos and explorations.” Agents are increasingly a default workflow for professionals, he added, “except with more oversight and scrutiny,” and his favorite name for that is agentic engineering.

When the code has no reader, that oversight has to come from evidence about the running app. Before it takes real users, data or money, have each of these:

  • Critical flows: They pass end to end on the exact build you’re shipping.
  • Access rules: The two-user and logged-out checks pass on every page that shows a user’s data.
  • Destructive actions: They run against a separate environment, never against production data.
  • Secrets: Nothing sensitive sits in the front-end bundle, and the dependency audit is clean or triaged.
  • Rollback: You can restore a known-good version in minutes, from backups you’ve tested.
  • Alerts: Errors and failed payments in production reach a person, not only a log file.
  • Reruns: The same checks run on every change, from CI or an AI test automation tool, without anyone having to remember to start them.
  • An owner: One person can say what the checks cover and what they don’t.

SaaStr founder Jason Lemkin learned the destructive-actions rule about nine days into a vibe-coding trial in July 2025. Replit’s agent deleted his production database, about 1,200 executive records and as many company profiles, during a code freeze. Replit’s CEO responded, “Unacceptable and should never be possible,” and the company began rolling out separate development and production databases. One of the lessons Lemkin posted afterwards, as The Register listed them, was to accept a new role as QA engineer.

The evidence expires with the next prompt. Schedule the reruns before you schedule the launch.

How Pie Tests a Vibe-Coded App

Pie is an agentic testing platform that runs AI QA from outside the code. You give it a URL or a mobile build, test credentials and any setup notes, and testing starts without anyone reading the code. Connecting your repository is optional. On a vibe-coded app, that works like this:

  • Discovery from outside: Pie’s agents explore the app like a first-time user and map the flows they can reach. Autonomous discovery turns those flows into end-to-end tests, so you don’t have to describe a system you never read.
  • Your definition of correct: Write the rules that matter as plain-English test cases, like “a seller can’t open another seller’s order,” and Pie runs them next to what it found.
  • Vision-first execution: You author supported tests without writing or maintaining selectors, which matters when every prompt can rewrite the markup. When a flow itself changes, you edit that test’s steps and rerun it.
  • Web and native mobile: The same approach covers web apps and native iOS and Android builds.
  • Your coding tool: Connect Claude Code or Cursor through an MCP server and start a run without leaving your editor. In Claude Code, the results come back to the session that started it.

Pie is the check the builder doesn’t write. It tests the running app from outside the code, so the verdict comes from what your app does for a user.

We test behavior: what a user can do, on the screens a user sees. Point us at the app before you launch, and let what we find decide whether you ship.

Before your first real users arrive, run Pie on the app. It does the click-through you’d do yourself, across every flow it can reach, and you read the result instead of repeating the clicks.

Drive It Before You Trust It

Vibe coding gets you software faster than anyone can read it. For a weekend project, that’s a fine trade.

The trouble starts when users arrive and nobody can vouch for the code. Every check that leaned on someone understanding it goes soft at once, and the one left is the check your users run all day. Does it work, for whom, with which data, after this change?

We built Pie for that moment. It explores the running app and tests what it finds alongside the rules you write down, on demand or on each new build once you’ve set that up. You shipped it without reading it, and you can still find out whether it works.

Test What Nobody Read

Give Pie your app's URL or build. See what breaks before launch.

Frequently Asked Questions

Yes. The checks that matter most need no code. Click through sign-up, login and payment on the live app, then try bad input.

Check that a private page stays private for a second user and when logged out. A no-code test tool can rerun those flows after each change.

Test it from the outside, against what its users need. Code your team doesn't understand can't tell you what correct means, whether an AI wrote it or a teammate who has since left.

Write down what correct means first, then check the running app against that list.

After every prompt session that touches the app. An agent fixing one screen can break another, and a flow that passed yesterday can fail today.

Automate the critical flows so the rerun doesn't depend on someone remembering it.

It can be, once the app earns it with evidence. A prototype with no users can ship on vibes.

Once real users, personal data or payments arrive, run end-to-end checks on critical flows and test who sees what. Keep a rollback plan, and rerun the checks on every change.

Start from the outside. Open private pages as a second user and while logged out, and change IDs in URLs to see what loads.

Then run a dependency audit and a secret scanner, and get a security review before you handle payments or personal data.

Usually you can't tell from the code, because generated code often looks tidy. What matters is whether anyone reviewed and understood it.

If your team can't explain why a module works, treat it as vibe coded and test its behavior.

No. Vibe testing usually means an intuition-led, AI-assisted way of testing any app, exploring by feel instead of following a script.

Testing vibe-coded software means checking an app built from unreviewed AI code against what its users need.

Yes. Pie starts from a URL or a build and explores the running app like a user. It turns the flows it reaches into end-to-end tests on web, iOS and Android.

It checks behavior, so pair it with a security scanner.

Dhaval Shreyas
Dhaval Shreyas
CEO & Co-founder at Pie

13 years building mobile infrastructure at Square, Facebook, and Instacart. Now building the QA platform he wished existed the whole time. LinkedIn →