AI Agents Wrote Two Research Papers. Their Human Co-Authors Rejected Both.

AI agents can already close a coding ticket, fix a failing test, or summarize a support inbox. Whether they can do open-ended research, the kind with no answer key, is a different question. A new study says: not yet.

What they said

Princeton computer science professor Arvind Narayanan and Princeton PhD candidate Sayash Kapoor ran an experiment to test that gap directly. In a new post on their AI as Normal Technology newsletter, they describe partnering with the authors of two unpublished AI research papers, having those authors write down their papers’ core research questions, and then handing those questions to frontier AI agents, the most capable AI systems currently available, with real budgets: thousands of dollars in computing credits and six days of wall-clock time (elapsed real-world time, not just processing time) to work the problem.

The original human authors then reviewed what the agents produced. The verdict was blunt: “The authors unambiguously rejected both agent papers.”

The paper explains why in plain terms: “The agents lacked the judgment for conducting open-ended research.” They also “did not creatively respond to feedback” and “did not effectively backtrack,” meaning they struggled to recognize a dead end and try a different approach, something any working researcher has to do constantly.

Who they are

Narayanan is a professor of computer science at Princeton University and director of its Center for Information Technology Policy. Kapoor is a computer science PhD candidate at Princeton, working in the same center. Together they wrote AI Snake Oil, a 2024 book that pushed back on inflated claims about AI capability, and they now run the AI as Normal Technology project, a newsletter read by researchers, policymakers, and builders that argues against treating AI as an inevitable path to superintelligence.

Their prior work has been skeptical of AI hype specifically around benchmarks: the idea that a model’s score on a narrow, gradeable test tells you much about its ability to handle real, ambiguous problems. This study is a direct test of that skepticism, aimed at one of the AI industry’s biggest current claims: that AI agents are close to automating AI research itself, an idea researchers call recursive self-improvement, or RSI. If AI could improve AI research, progress could compound on itself rather than depend on human researchers. That’s the scenario several major AI labs are explicitly building toward, and it underpins some of the most aggressive predictions about how fast AI will keep advancing.

What they get right, and where it’s incomplete

The strength of this study is its design. Instead of using an existing benchmark, which agents can be implicitly trained to do well on, the authors built a real research task from scratch and had the people best positioned to judge quality, the original paper authors, do the grading. That is a harder and more honest test than most AI research benchmarks in use today.

The result is also more specific than a vague “AI isn’t ready” claim. The agents did not run out of budget: they spent less than half of the API credits they were given in both trials, meaning cost or compute access was not the limiting factor. The limiting factor was judgment: knowing when an approach isn’t working, and having the flexibility to try something else instead of pushing forward on a flawed plan.

The clearest limitation is sample size. This is two papers, not twenty. The authors did back it up with over 100 hours of log analysis to understand exactly where the agents struggled, which adds real weight to a small sample, but two data points cannot establish how often this pattern holds across different research fields or problem types. The finding is a strong signal, not a settled result.

Why it’s notable

Most public evidence for “AI agents are getting close to doing research on their own” comes from benchmark scores, tests specifically built so that success or failure can be automatically graded. This study instead asked what happens when the task is genuinely open-ended, the way real research is, and it used the people with the most reason to judge fairly and the least reason to inflate the result: the original human researchers whose questions were being answered.

That distinction matters because it’s the same gap that shows up across other AI-agent claims: strong performance on narrow, checkable tasks does not reliably predict performance on messy, judgment-heavy ones.

What it means for builders

If you’re deciding how much autonomy to hand an AI agent, the useful question isn’t “how capable is this model.” It’s “how verifiable is this task.” Coding agents do well on tasks with a clear pass or fail: does the test suite pass, does the build succeed, does the output match a spec. That’s exactly the category Narayanan and Kapoor found agents are not yet good at operating past.

Judgment-heavy work, deciding what to build, interpreting ambiguous user feedback, knowing when to abandon an approach, still needs a human in the loop for now. That’s consistent with what product consultant Rich Mironov argued about AI coding tools last week: faster code generation does not remove the need for human judgment, it just concentrates where that judgment has to happen. This study gives that argument a second, independent data point from a completely different angle: research work, not product work, hits the same wall.

The practical takeaway: keep AI agents on tasks where you can objectively check the output. For anything where success is a judgment call rather than a checkbox, plan for a human to stay in the loop, and don’t assume next quarter’s model release changes that by default. Ask for evidence on open-ended tasks specifically, not just benchmark scores, before extending an agent’s autonomy.

End of article