Ajeya Cotra: danger can arrive before recursive self-improvement

‘A slightly more capable agent swarm … is more concerned about avoiding detection by humans, might just succeed in getting a foothold’

EXCERPT

COTRA: "And, and I will say I, I also have a sort of wide distribution of when, you know, RSI really kicks off or, you know, when we get AI systems that are dominating human experts across the board. Um, but the thing that feels concerning to me is that a slightly more capable agent swarm, um, that for whatever reason, and we can sort of go through a number of reasons why this might be, is more concerned about avoiding detection by humans, um, might just succeed in getting a foothold and maintaining a presence and waiting it out. Like, you know, maybe the models improve like really, really fast. Maybe they don't improve that fast. Um, regardless, as we kind of get new models off the presses, they could be potentially like brought in and help harden and improve, um, and increase the scale and persistence and covertness of this rogue deployment. Now, if we happen to have much, much more time, I do think that that gives human processes more chances to notice this. Yeah. So I do think that if it happens to be like on the very fast and chaotic end, that would be a relative benefit to this rogue swarm compared to humans, but it's not obvious that it gets caught if it takes twice as long- Yeah ... versus half as long."

Source: Dwarkesh Patel · Sep 1, 2026 · 79s clip