What changed
Replit added a new way to check a built app for security holes: a black-box penetration test that probes your live, running app the same way an outside attacker would, instead of just reading your source code for mistakes.
A penetration test, or “pen test,” is a safe, authorized practice attack. Someone (or in this case, an AI agent) tries to break into your app on purpose so you can fix what they find before a real attacker does. Replit already runs a code-reading scan, where an agent reads through your app’s source code looking for patterns that tend to be dangerous, like a missing permission check. That runs automatically every time you build. The new scan is different: it doesn’t look at your code at all. It clicks through your live app in a browser, watches the network traffic to figure out what features exist, and then tries to find a way in, the same starting point a real hacker would have.
Replit’s Security Center now offers three scan levels. Level 1 is free and runs dependency checks (looking for known flaws in the outside code libraries your app depends on) plus a basic code scan. Level 2 is a deeper scan from the code-reading agent. Level 3, the new option, runs the code-reading scan and the new black-box scan together. You start it manually from your project’s Security Center. Replit’s post doesn’t say which plan you need for Level 2 or Level 3, or how long a Level 3 scan takes to finish.
The black-box scan runs twice: once with no login, to see what a random visitor could reach, and once logged in as a real user, to check whether that user can reach things they shouldn’t (a problem called privilege escalation, where someone gets more access than they’re supposed to have). It all happens against a private sandbox copy of your app, not your live production version, so the test itself can’t cause real damage or lock out real users.
Why it matters
Reading code and attacking a live app turn up different problems, and Replit’s own post says the two rarely find the same thing. The code-reading scan is better at catching logic flaws, like a user whose access was revoked in the database but who can still log in because a check elsewhere in the code was never updated. The black-box scan is better at catching exposure flaws, like an admin dashboard that’s reachable by anyone who guesses the URL, with no login screen stopping them.
That gap matters because code can look correct in isolation and still be exposed once it’s actually running. A permission check might exist in your code but never actually run if that page was connected to the wrong web address behind the scenes, or if a debug page got left reachable in a settings file the code scanner never touched. Those problems only show up when something actually tries to reach the running app from the outside, which is exactly what the black-box scan does and the code scanner doesn’t.
The concrete risk: an app can pass every code-level scan and still be sitting open to anyone who finds the right URL. If you’re building something that stores real user data, treating a clean code scan as proof the app is safe is the mistake to avoid. Run the black-box pass too, and look specifically at anything it flags that the code scan didn’t, since those are the issues a real visitor could reach today.
What this reflects
This follows a pattern building across AI app builders. On August 6, Lovable added an automatic trust page to every published app, showing platform security controls like encryption and dependency scanning without any setup. Replit’s move goes a step further: instead of just reporting on configuration, it actively tries to break in. Both point at the same shift. As more people ship real, paying apps through AI builders instead of just prototypes, these platforms are competing on whether they can catch the kind of security holes that used to require a paid outside firm and weeks of back-and-forth, not just on how fast they can generate an app.
Who should care
This matters most if you’ve published, or plan to publish, a Replit app that stores real user data or handles logins, especially anything you’re charging for or sharing outside a small test group. It matters less for a prototype with fake data that no one outside your own testing has touched, though it’s still worth running before you add your first real user.
What builders should do next
Pick a project you’ve already built and run both a Level 2 scan and a Level 3 scan on it back to back. Compare two things: how many total findings each one returns, and whether the Level 3 pass surfaces anything the Level 2 pass didn’t, especially anything involving login screens, admin pages, or user permissions. Replit’s own post says overlap between the two is rare, so if Level 3 comes back with zero new findings, that’s a real signal your app’s live behavior matches its code. If it surfaces something new, particularly an exposed page or an access check that doesn’t hold up, fix that before you treat the app as safe to hand to real users. A scan that only reads code can tell you the code looks right. It can’t tell you what’s actually reachable.
End of article