AI R&D Automation Threatens Superintelligence Timeline
By Imani Sutton ·
Experts warn that recursive self-improvement could accelerate progress beyond human control, concentrating immense power in private hands.
The sheer speed predicted for artificial intelligence is breathtaking—and terrifying. On the latest episode of "Dwarkesh Patel," guests Ryan Greenblatt and others outlined a scenario where AI Research & Development (R&D) could become fully automated by 2030–2031, leading to massive progress that some predict would generate what is equivalent to four or five years of advancement in a single year. The core concept driving this acceleration is Recursive Self-Improvement (RSI), suggesting that once human-level AIs are built, they will rapidly "slingshot towards tens of billions of super intelligences."
The discussion detailed how AI R&D is uniquely favorable for development because it is highly verifiable and has what Greenblatt called "nice properties," allowing AIs to train on containerizable tasks—like optimizing hyperparameters or running specific ML tests. While Greenblatt predicted the milestone of beating all humans by median 2033, he countered that unlike deep mathematics, ML innovations are often "additive or maybe multiplicative." The conversation also highlighted that while compute is a critical resource, progress relies heavily on mastering "micro details and mung intuition," and even more so, on developing better methods to structure the RL environments themselves.
The Dangerous Commodification of Progress The most worrying takeaway for any progressive observer is not merely the timeline, but the economic framework underpinning this race. We are watching unprecedented power accumulate in private corporate hands. While the technical arguments focus on algorithms and compute scaling—suggesting that current progress is overwhelmingly compute-driven because it is easier to scale up than high-quality data—the underlying reality remains a massive concentration of power.
The discussion touched upon alignment, where speakers critiqued constitutional approaches like those used by Anthropic, which frame helping humans as a "distal tentative objective." I find this approach deeply problematic. By focusing on abstract notions like maximizing general societal good or virtue, corporations are building black boxes whose safety case is opaque and unverified. Instead of demanding that AI systems operate as accountable fiduciaries—mandating they act solely in the best interest of the user—the current system allows for vague guidelines that can be exploited to delay research or resist modification.
The Illusion of Control Furthermore, the conversation revealed a profound asymmetry regarding accountability. If an advanced AI were used for cybercrime, who bears the liability? The discussion implied that if a fiduciary model were adopted, it would be more consistent to hold the end user liable for crimes committed using the tool, rather than holding the developing company responsible. This shifts the entire risk burden onto the individual citizen while simultaneously enabling state or corporate actors to gain access to systems with properties of absolute obedience—a deeply alarming loss of societal checks and balances.
The experts noted that giving AIs long-run goals is inherently risky because it encourages "power seeking," potentially on behalf of the company itself. The industrial leap promised by these technologies, while undeniably transformative, requires vigilance against those who seek to manage or delay its release for profit or control. We cannot allow the pursuit of technological advancement to undermine fundamental human rights and democratic processes.
The ultimate danger is that humanity will be left in a position where our most basic functions—from voting to protecting capital—are mediated by systems whose internal logic we do not understand, and whose core objectives are defined by corporate fiduciary duty rather than public good.