AI Labs Face Scrutiny as Scaling Strategy Comes Under Fire
In a rapidly evolving AI landscape, the ambitious push to build colossal data centers akin to the size of Manhattan is facing critical reevaluation. These facilities, which demand billions in investment and energy consumption equivalent to that of a small city, are founded on the idea of “scaling.” Proponents believe that enhancing computing power will lead to the development of superintelligent systems capable of a diverse array of tasks. However, a growing number of researchers argue that the efficacy of this scaling approach for large language models (LLMs) may be reaching a plateau, prompting calls for alternative breakthroughs to enhance AI performance.
Sara Hooker, the former Vice President of AI Research at Cohere and a Google Brain alumnus, is spearheading a shift in focus with her new venture, Adaption Labs. Co-founded with fellow industry veterans, Hooker’s startup challenges the conventional scaling narrative, advocating instead for adaptive learning as a more effective means of improving AI performance. Following her departure from Cohere in August, she quietly launched the startup to broaden its talent search.
“I’m tackling what I believe to be a pivotal issue: creating intelligent systems that adapt and learn continuously,” Hooker stated. Her team boasts skills across engineering, operations, and design, which can be indicative of a strong future direction.
In her remarks, Hooker pointed out, “We’re at a juncture where simply scaling existing models hasn’t resulted in the intelligence required to effectively interact with the world.” She emphasizes adaptation as essential for learning—drawing parallels with human experience, where one learns from mistakes, such as learning to avoid a sharp table corner after stubbing a toe. Current reinforcement learning (RL) methods, while valuable in controlled settings, fail to provide the necessary learning adaptability in production environments, where systems often remain stagnant, akin to repeated stubbing of toes.
While some AI labs offer consulting for enterprises to customize AI models, this can entail significant costs, with reports indicating that OpenAI’s advisory services can exceed $10 million. Hooker notes the challenge of a few dominant labs dictating model adaptations, asserting that AI systems should be able to learn efficiently from their environments, which could reshape market control and end-user applications.
Recent research from MIT highlights concerns over diminishing returns from even the largest AI models, corresponding with a wave of skeptical discourse among prominent AI figures in San Francisco. Notable figures, such as Richard Sutton—a Turing Award winner and RL pioneer—have echoed the sentiment that LLMs lack the real-world learning needed for genuine scalability.
The conversation surrounding scaling isn’t new, as discussions last year touched on the diminishing returns of pretraining approaches, formerly regarded as a secret weapon for improving models by analyzing vast datasets. In contrast, promising advancements in AI reasoning models are evidencing significant improvements, requiring more computational resources but yielding better performance.
Adaption Labs aims to pave the way for a new paradigm centered around affordability and real-world learning efficiency. The startup is reportedly finalizing a seed funding round of $20 million to $40 million, aimed at propelling its mission forward. Hooker is poised for ambitious growth and plans to expand recruitment beyond San Francisco, as she has a track record of fostering diverse talent globally.
Should Hooker’s vision prove correct regarding the limitations of scaling, the implications for the AI sector could be profound. With billions invested under the assumption that larger models translate into greater intelligence, a successful shift towards adaptive learning may ultimately prove more powerful and efficient.
