Artificial Intelligence May Already Be Plateauing

According to a recent story from The Information, OpenAI’s next-generation Orion model of ChatGPT, which is both rumored and denied to be arriving before the end of the year, might not be everything that has been hailed when it does.

According to the report, which quotes unnamed OpenAI staff, the Orion model has shown a “far smaller” improvement over its predecessor, GPT-4, than GPT-4 did over GPT-3. According to the same individuals, Orion “isn’t reliably better than its predecessor [GPT-4] in handling certain tasks,” particularly coding applications, even though the new model is noticeably better at general language functions like creating emails or summarizing documents.

According to the Information report, one of the main reasons for the new model’s negligible improvements is the “dwindling supply of high-quality text and other data” that may train new models. Simply put, the AI sector is rapidly approaching a training data bottleneck after removing the simple sources of social media data from websites like X, Facebook, and YouTube (the latter twice). Pre-release training is being slowed significantly because of these organizations’ growing inability to identify the kinds of complex coding problems that will help them push their models beyond their existing limits.

This decreased training effectiveness has significant business and environmental ramifications. The amount of energy, water, and other resources is predicted to increase sixfold over the next ten years as frontier-class LLMs expand and continue to push their parameter counts into the high trillions. Because of this, the country’s current power infrastructure cannot supply the power required for their expanding networks of artificial intelligence data centers. For this reason, Google is purchasing the output of seven nuclear reactors, AWS is purchasing a 960 MW plant, and Microsoft is attempting to restart Three Mile Island.

To get around the shortage of suitable training data, OpenAI has established a “foundations team.” These methods might entail the use of artificial training data, like that produced by Nvidia’s Nemotron family of models. The group is also investigating ways to enhance the model’s performance after training.

Initially believed to be the code name for OpenAI’s GPT-5, Orion is now expected to arrive in 2025. It’s unclear if we’ll have enough electricity to see it through to completion without causing our municipal electrical grids to brown out.