Is the Tech Industry
Already on the Cusp of an A.I. Slowdown?
Companies
like OpenAI and Google are running out of the data used
to train artificial intelligence systems. Can new methods continue years of rapid
progress?
Demis Hassabis, one of the most influential artificial
intelligence experts in the world, has a warning for the rest of the tech industry:
Don’t expect chatbots to continue to improve as quickly as they have over the last
few years.
A.I.
researchers have for some time been relying on a fairly simple concept to improve
their systems: The more data culled from the internet that they pumped into large
language models — the technology behind chatbots — the better those systems performed.
But
Dr. Hassabis, who oversees Google DeepMind, the company’s
primary A.I. lab, now says that method is running out of steam simply because tech
companies are running out of data.
“Everyone
in the industry is seeing diminishing returns,” Dr. Hassabis
said this month in an interview with The New York Times as he prepared to accept
a Nobel Prize for his work on artificial intelligence.
Dr. Hassabis is not the only A.I. expert warning of a slowdown.
Interviews with 20 executives and researchers showed a widespread belief that the
tech industry is running into a problem that to many was unthinkable just a few
years ago: They have used up most of the digital text available on the internet.
That
problem is starting to surface even as billions of dollars continue to be poured
into A.I. development. On Tuesday, Databricks, an A.I. data company, said it was
closing in on $10 billion in funding — the largest-ever private funding round for
a start-up. And the biggest companies in tech are signaling
that they have no plans to slow down their spending on the giant data centers that run A.I. systems.
Not
everyone in the A.I. world is concerned. Some, like OpenAI’s
chief executive, Sam Altman, say that progress will continue at the same pace, albeit
with some twists on old techniques. Dario Amodei, the
chief executive of the A.I. start-up Anthropic, and Jensen Huang, Nvidia’s chief
executive, are also bullish.
The
roots of the debate trace back to 2020 when Jared Kaplan, a theoretical physicist
at Johns Hopkins University, published a research paper showing that large language
models steadily grew more powerful and lifelike as they analyzed
more data.
Researchers
called Dr. Kaplan’s findings “the Scaling Laws.” Just
as students learn more by reading more books, A.I. systems improved as they ingested
increasingly large amounts of digital text culled from the internet, including news
articles, chat logs and computer programs. Seeing the raw power of this phenomenon,
companies like OpenAI, Google and Meta raced to get their
hands on as much internet data as possible, cutting corners, ignoring corporate
policies and even debating whether they should skirt the law, according to an examination
this year by The Times.
It
was the modern equivalent of Moore’s Law, the oft-quoted maxim coined in the 1960s
by the Intel co-founder Gordon Moore. He showed that the number of transistors on
a silicon chip doubled every two years or so, steadily increasing the power of the
world’s computers. Moore’s Law held up for 40 years. But eventually, it started
to slow.
The
problem is, neither the Scaling Laws nor Moore’s Law is an immutable law of nature.
They’re simply smart observations. One held up for decades. The others may have
a much shorter shelf life. Google and Dr. Kaplan’s new
employer, Anthropic, cannot just throw more text at their A.I. systems, because
there is little text left to throw.
“There
were extraordinary returns over the last three or four years as the Scaling Laws
were getting going,” Dr. Hassabis said. “But we are no
longer getting the same progress.”
Dr. Hassabis said that existing techniques would continue to improve
A.I. in some ways. But he said he believed that entirely new ideas were needed to
reach the goal that Google and many others were chasing: a machine that can match
the power of the human brain.
Ilya
Sutskever, who was instrumental in pushing the industry
to think big as a researcher at both Google and OpenAI
before leaving OpenAI to create a start-up this spring,
made the same point during a speech last week. “We’ve achieved peak data, and there’ll
be no more,” he said. “We have to deal with the data that we have. There’s only
one internet.”
Dr. Hassabis and others are exploring a different approach. They
are developing ways for large language models to learn from their own trial and
error. By working through various math problems, for instance, language models can
learn which methods lead to the right answer and which do not. In essence, the models
train on data that they themselves generate. Researchers call this “synthetic
data.”
OpenAI recently released a new system, called OpenAI o1, that was built this way. But the method works only
in areas like math and computing programming, where there is a firm distinction
between right and wrong.
Even
in these areas, A.I. systems have a way of making mistakes and making things up.
That can hamper efforts to build A.I. “agents” that can write their own
computer programs and take actions on behalf of internet users, which experts see
as one of A.I.’s most important skills.
Sorting
through the wider expanses of human knowledge is even more difficult.
“These
methods only work in areas where things are empirically true, like math and science,”
said Dylan Patel, chief analyst for the research firm SemiAnalysis,
who closely follows the rise of A.I. technologies. “The humanities and the arts,
moral and philosophical problems are much more difficult.”
People
like Mr. Altman of OpenAI say that these new techniques
will continue to push the technology ahead. But if progress reaches a plateau, the
implications could be far-reaching, even for Nvidia, which has become one of the
most valuable companies in the world thanks to the A.I. boom.
During
a call with analysts last month, Mr. Huang, Nvidia’s chief executive, was asked
how the company was helping customers work through a potential slowdown and what
the repercussions might be for its business. He said that evidence showed there
were still gains being made, but that businesses were also testing new processes
and techniques on A.I. chips.
“As
a result of that, the demand for our infrastructure is really great,” Mr. Huang
said.
Though
he is confident about Nvidia’s prospects, some of the company’s biggest customers
acknowledge that they must prepare for the possibility that A.I. will not advance
as quickly as expected.
“We
have had to grapple with this. Is this thing real or not?” said Rachel Peterson,
vice president of data centers at Meta. “It is a great
question because of all the dollars that are being thrown into this across the board.”