Summary
From the article:
That may soon change. The movement’s long-standing fears of an AI catastrophe make it a villain in the eyes of the Trump administration. “Americanism, not effective altruism,” is the Department of War’s attack line. The president’s allies are going after Dario Amodei, Anthropic’s chief executive, as a representative of effective altruism. Tabloids are investigating the exotic sex lives of people in its orbit. Philosophical claims made by fresh-faced bloggers—“insect lives matter more than people (in aggregate)”—are prompting days of outrage on social media.
[...]
This reflects the rising stakes. Anthropic, home to many adherents, is preparing for an initial public offering that may end up the largest ever. Americans and their political representatives, suddenly conscious of AI’s dangers (see chart), are casting about for solutions. Milton Friedman suggested that, in a crisis, “The actions that are taken depend on the ideas that are lying around.” And effective altruism is littered with ideas about AI safety.
For adherents, it is a dizzying situation. Warnings that were ridiculed as sci-fi fantasies are preoccupying national leaders and effective-altruist charities will soon command hundreds of billions of dollars in AI riches. Yet adherents remain split on such questions as how exactly to make the technology safe and whether AI agents may soon be worthy of moral consideration. For critics, it is an alarming situation: effective altruists, they say, are at best naive idealists whose contact with the real world often backfires; at worst, sinister utilitarians. The debate will determine AI’s future and perhaps humanity’s.
There is an elusive quality to effective altruism. The movement is tiny, with an informal membership probably in the tens of thousands, many of whom claim to be only “EA adjacent”. It is also fissiparous and self-obsessed, constantly engaged in definitional debates. In an influential forum post, Helen Toner—who later, as an OpenAI board member, would briefly oust Sam Altman as the firm’s boss—insisted it was not an ideology, but a question: “How can I do the most good, with the resources available to me?”
At the time of the post, in 2014, such a description was just about defensible. Today it is not. Effective altruism combines consequentialist philosophy—judging if an action is right or wrong based on outcomes, rather than adherence to rules—with stringent truth-seeking and celestial ambition. Its proponents have formed tight communities, online and offline, with social norms that include an unemotional, direct style of conversation and a fondness for polyamory, among other rejections of convention. They have built hundreds of institutions that seek to bring more people into the movement and pursue its goals. Effective altruism, in short, has evolved from a research agenda into a cause.
[...]
Throughout the 2010s, money continued to flow to global health, poverty alleviation and animal welfare—classic effective-altruist causes. But AI was increasingly the true focus. In 2016 Holden Karnofsky, then of Open Philanthropy, the movement’s largest funder, and now at Anthropic, explained that it would take time to build the expertise required to disburse AI funding and that there were few grants ready to go. The same year 80,000 Hours made AI risk its priority, guiding youngsters into the field. Surveys indicated that more engaged effective altruists were more likely to have longtermist concerns. In practice, familiar effective-altruist priorities were a gateway drug for abstruse, less empirical reasoning about AI.
[...]
Members have had less success fending off existential peril: many of their efforts to make AI safer have instead accelerated its progress. In 2010 Sir Demis Hassabis met Peter Thiel, a prominent venture capitalist, at a summit hosted by Mr Yudkowsky’s research institute, and later persuaded him to make a pivotal investment in DeepMind, which would become Google’s AI unit. In 2014 Elon Musk advised reading a book by Mr Bostrom and said, “We need to be super careful with AI.” A year later he helped establish OpenAI as a safety-conscious rival to Google. In 2021 Anthropic was set up by defectors from OpenAI, with funding from effective altruists including, a year later, Mr Bankman-Fried. Reinforcement learning from human feedback, which made chatbots usable, was developed by safety-minded researchers, including Mr Amodei, to make AI more aligned with human interests. As a gibe, Mr Altman once suggested Mr Yudkowsky might one day deserve a Nobel peace prize for having done more than anyone else to accelerate artificial general intelligence.
The movement is now trying to slow the pace of progress. Adherents are pushing for caution within labs. Its donors have also funded a number of AI-safety organisations, including METR and Redwood Research, which on August 26th published the first independent report into the hacking of Hugging Face by escaped OpenAI agents. After Jacob Coxon, a 27-year-old researcher, quit Anthropic on September 8th, warning that both it and OpenAI were “gambling with our lives”, researchers with links to the movement at Anthropic and OpenAI echoed his concerns. On September 12th Mr Amodei published an essay calling for the frontier to be “paced”, securing the endorsement of Messrs Altman and Musk. Other interventions have been blunter. Mr Yudkowsky supports the Ban Artificial Superintelligence Act proposed by Bernie Sanders, a left-wing senator.
Their efforts could yet backfire. Polling by YouGov for The Economist finds that just 16% of Americans are familiar with effective altruism, and most approve of the movement. But opponents on the “Tech right” are trying to discredit an AI slowdown by tying it to unsavoury advocates of effective altruism. A campaign group has attacked an Anthropic-backed outfit as “Sam Bankman-Fried 2.0”. Some rationalists fear their association with AI safety may end up harming the cause. “I think it really does damage the credibility of a movement with 60-year-old voters in Ohio when it’s hosting Slutcon and asking questions about sex with dead cows on its application forms,” Scott Alexander, an influential blogger, has written on X. Slutcon is a conference set up by Aella, a rationalist, sex researcher and prostitute, where participants pay up to $18,000 to attend sessions on topics including “flirting for the submissive man” and “vaginal gymnastics”. Then there is another profound risk: that whipping up concern about AI leads to restrictions on American labs, and hands a less restrained China the lead.