Summary
From the article:
In the weeks leading up to the initial public offering, employees of Anthropic — who affectionately call themselves “Ants” — have been preoccupied with concerns about how to slow down A.I. Following a recent spate of rogue agent episodes, the industry has felt doomier than ever.
Even the most ardent techno-optimists have quieted their cheers. Group chats and gatherings instead debate pacing the frontier: how the leading A.I. labs can stop racing and agree to cooperate on the speed of model development, ensuring they have time to make the models safe before releasing them to the world.
For many Ants, the hard part is just beginning. They see recursive self-improvement, in which current A.I. models autonomously build the next generation, looming on the horizon. Because of this potential breakthrough, Anthropic’s co-founder Jack Clark as well as the independent forecasting group A.I. Futures Project have moved up the likely arrival date for superhuman A.I. researchers. They now anticipate it being in 2027.
[...]
I have spoken to dozens of Ants during the past two years, from company founders to young researchers to jaded ex-employees. I have found them surprisingly earnest in their dedication to making A.I. go well. But their concerns are also idiosyncratic and insular, often far removed from the concerns that most people hold about A.I.
They do not view Anthropic as a normal for-profit company. So as the U.S. economy and our personal financial fates become yoked to the Ants, it’s critical that we understand what they believe.
Juxtaposed with what they presume will be the imminent arrival of superintelligence, the upcoming payday seems parochial to many at Anthropic. Some employees describe the I.P.O. with a shrug, likening it to just another funding round.
This is a peculiar attitude for a crowd of young people who, in many cases, are about to become deca- and centimillionaires overnight. The Ants are not your average tech workers. They spend morning to night at rows of desks in the bare, high-security offices in San Francisco’s downtown, where the common books on display include Richard Rhodes’s “The Making of the Atomic Bomb” and paperback printouts of company writings, such as Mr. Amodei’s “Machines of Loving Grace” and the in-house philosopher Amanda Askell’s “Claude’s Constitution.”
No one is compelled to work the grueling “996” hours (9 a.m. to 9 p.m., six days a week) that have become common in the tech industry, but a good number of Ants do so anyway out of their own feverish devotion. Many cram themselves into modest group apartments, host house parties where few, if any, drink and blank when I ask how they spend their lavish salaries, even with my prompts: “A wardrobe update, maybe?” “Ever considered flying business class?”
[...]
The money is secondary. The most common form of I.P.O. talk involves discussions of where to allocate the newly liquid stock in their charitable donation accounts.
As a company, Anthropic has been strongly influenced by effective altruism, which holds the belief that the wealthy are obliged to share their fruits with the world. New hires are invited to pledge a percentage of their equity to charity when they start, which Anthropic matches part of. Top targets include charities dedicated to animal welfare, global health and research on A.I. alignment and safety. Many have begun giving big to political campaigns.
As for their own stock, few of the Ants I spoke to planned to sell right after the I.P.O. The notion of an “A.I. bubble” seems not to exist at Anthropic. Employees expect their equity to eventually be worth much, much more.
Anthropic crisply embodies the mind-rending paradox of contemporary A.I. The research scientists who’ve done the most to extend A.I. capabilities are often the most insistent about its risks. Anthropic is the most vocal company when it comes to economic impacts such as mass automation, yet it’s also the leading vendor of the A.I. enterprise agents that threaten to replace white-collar workers.
Such a stance can seem duplicitous and self-serving, both convenient and convoluted. “Our ability to have a positive impact on the world scales in relation to our reach and stature as a successful company,” reads one eye-roll-inducing statement in Anthropic’s I.P.O. prospectus. While the company’s leaders have been candid about the risks, they have been less forthcoming about their solutions. When asked why they should birth what they suspect is an existentially dangerous technology, their answer amounts to, essentially, just trust us because we’re the good guys.
[...]
In the summer of 2022, Anthropic decided not to make Claude, its A.I. chatbot, available to the public over internal concerns about safety. This allowed OpenAI — which had gotten wind of Anthropic’s internal prototype — to design a similar product, rush ChatGPT out the door first and gain a decisive lead in consumer mindshare.
In Anthropic company lore, this saga is put on a pedestal as an example of choosing principle over profit. Yet it may have also been a strategic wake-up call for Mr. Amodei, who called the sacrifice a “one-off.” Increasingly, Mr. Amodei seemed to see how being the leader in the A.I. race conferred compounding benefits in terms of revenue, recruiting, fund-raising and developing the next-generation A.I. models. Once commercial A.I. products had been introduced, Mr. Amodei seemed unwilling to settle for second place.
[...]
First, Anthropic assumes the A.I. race itself is inevitable, so no single actor can stop superintelligence from being built. The financial rewards for building a machine that can match or exceed humans’ cognitive abilities are so great that someone is guaranteed to pursue it. If not Mr. Amodei and his team, then Sam Altman, or Mark Zuckerberg or Elon Musk.
Second, Anthropic assumes it is the most responsible of A.I.’s possible stewards. In public and internally, Mr. Amodei likes to invoke OpenAI and the Chinese government as competitive boogeymen. These are viewed as actors that are pushing just as hard as Anthropic on building and commercializing powerful A.I. models, but that would be far less responsible — reckless at best, and authoritarian at worst. Looking at the recent evidence, the most severe rogue agent hacks have indeed come out of OpenAI. And if you think Claude is the A.I. with the best moral values, then expanding Claude’s market share could be construed as a virtue in itself.
Third, Anthropic assumes the “winner” of A.I. will gain tremendous influence over the A.I. ecosystem and the U.S. government. Through commercial success, Anthropic aims to show the industry that being safety-minded can be good for business, good for science and good for attracting top talent.
[...]
And why is Anthropic confident that its own power won’t corrupt? The company prides itself on costly signals: making choices that trade off against short-term profits, whether prohibiting Chinese entities from buying its products; placing 5 percent of its shares into a “Public Benefit Fund” to spend on “large donations to traditional nonprofits or direct distributions to specific populations” of its choice; or refusing to permit the Department of Defense to use Claude for domestic mass surveillance, an act that infuriated the Trump administration and briefly got Anthropic marked as a supply-chain risk last winter.
[...]
In internal debates, early employees act as informal ethos-enforcers, questioning whether their colleagues’ decisions are sufficiently “aligned.” It’s a company of loyalists: Both leaks and departures have been rare. Arguably, this focus has paid off. Anthropic has progressed quickly on automated coding while OpenAI, whose culture is understood to resemble a bunch of battling fiefs, was distracted by drama and side quests.
[...]
Some anxious Ants have begun asking whether Anthropic should stop training better frontier models now, even if it can’t guarantee that competitors like OpenAI will do the same. This suggestion was shut down by Mr. Amodei, who urged staff members to stop listening to outside activists and explained that the company could pace itself without shooting itself in the foot. It was all part of the same strategy: Anthropic must keep winning to keep the world safe, and even if it stops, its competitors will not.
[...]
Anthropic hopes its public benefit corporation status will protect it against normal company incentives. Under its five-tier stock structure, the founders, employees and trustees can elect board members, while traditional investors have a limited role. The company is also trying to secure a congressional antitrust exemption so that it can coordinate a slowdown with the other American A.I. labs.