ChatGPT celebrates its 2nd anniversary. And the GPT model behind this tool has already evolved several times. The latest launch, o1, does not really offer a more powerful version. Rather a tool that functions differently: it breaks down tasks, step by step, to carry out more complex reasoning operations. If for months, the director of OpenAI Sam Altman has repeatedly repeated that a new revolutionary model, surpassing the intellectual capacities of humans, will be released, the latest news is less encouraging.
The model, known internally as Orion, did not achieve the desired performance, stumbling on tasks of computer programming or reasoning. The progress is real but of a lesser magnitude than the previous leaps. OpenAI’s competitors also face diminishing returns. Google’s next version of Gemini fails to meet internal expectations. Anthropic had to postpone the release schedule of its highly anticipated Claude 3.5 Opus model.
These difficulties challenge the rule of scale, according to which more computing power, more data and larger models would inevitably pave the way for greater advances in the power of artificial intelligence. Dario Amodei, the founder of Anthropic, devoted the first twenty minutes of his five-hour speech to Lex Friedman’s podcast about this. The question is crucial. Containing hundreds of thousands of Nvidia chips linked together, the data centers needed to train the models are increasingly expensive. Dario Amodei mentioned the figures of 1 billion dollars today, a few billion in 2025, around ten billion in 2026. It will therefore be necessary to justify such investments.
Getting around the AI wall
The main limitation these companies face is the availability of high-quality untapped sources for training. It is easy to generate quantities of data synthetically but they are not diverse enough. However, repetitive or biased content reduces performance. According to some estimatesall available public texts could be exhausted between 2026 and 2032. Currently, around 33% of quality public data has already been used to train recent models. OpenAI has signed agreements with content publishers to feed the training of AIs before their launch, with exclusive sources. These efforts are slower and more expensive than simply scraping the web.
Beyond pre-training, one of the paths to progress is “post-training”. This has long been based on human feedback that helps improve responses and refine how the model should interact with users. But now, companies are hiring high-level specialists who can qualify data related to their area of expertise, whether it’s mathematics or coding. The new models take an approach called “expert mixing” which involves assembling distinct subnetworks. Each is specialized in a subset of input data, in order to jointly perform a task.
As the leaders progress more slowly, competing players, particularly in China, are catching up. Chinese society 01.aifounded by entrepreneur Kai-Fu Lee, claims having created a model equivalent to GPT-4 using 30 times less computing power. Last week, Deepseek, a spin-off from hedge fund High-Flyer Capital Management, revealed its complex reasoning model R1 which outperforms OpenAI’s o1.
However, it is too early to declare that the generative artificial intelligence bubble will burst. The speed of inference – the processing of information by the model to generate a response – is progressing rapidly, making it possible to break down tasks more finely and therefore to have better reasoning. Many new uses will appear and develop, even with models whose performance is growing less quickly than in the past. This is particularly the case for agents, these specialized models which connect different data sources to understand a context and carry out coherent tasks with a low risk of errors.
.