Empires of AI by Karen Hao thoughts

• It's unsurprising that Sam Altman is such a smarmy power-hungry sociopath.
• It's interesting how they do this stupid XKCD #605 linear extrapolation of computing needs, and instead of looking into all the ways they could try to reduce their computational requirements, and making that their mission to decrease those requirements — since the human brain, the most intelligent thing we know of, requires very little power, so surely part of the route to practicable "AGI" would be low power requirements? and even if we require more and more compute to make computers more intelligent, reducing those requirements by a large base factor and making them scale less surely opens up a lot more headroom to develop and improve, so wouldn't that be the obvious option? — they decide this means they need to immediately sell out their basic principles and cozy up to a megacrop.
• In general, the early parts of the book are just a story of OpenAI one by one selling out its principles in exchange for convenience and power.

• It's really interesting how conveniently self-serving OpenAI's founding mythology is:

  • Inevitabilism — the assumption that if they don't invent AI, someone else will, and they have to do it first (see the next point) — allows them to absolve themselves of meaningful ethical responsibility or questions about what kind of future they should make, at least on the broad questions, even though in theory their entire purpose as a company is to ensure a better future.
  • Exceptionalism — the assumption that they're the best possible stewards for AI — allows them to ignore regulations, be secrative to get ahead, and motivates their desire for more centralized power and control; when combined with the aforementioned inevitabilism, it gets even worse, as it implies that they have to beat other people working on AI to the punch, justifying them to accelerate massively without having to think about how to do anything sensibly, sustainably, efficiently, or carefully, and without having to e.g. properly communicate with the public and consult academics and lawmakers and the public.
  • "Scaling Laws" as absolute. They're not wrong that scaling is the easiest way to increase model capabilities when you have almost infinite money, but combined with the previous two points, it justifies a really dumb, blind, hurtling-toward-the-edge-of-the-cliff mindset of scaling at all costs.
• Hao's descriptions of disaster capitalism and the exploitation of data annotation workers is poignant, and really makes me think hard about my support of the creation of LLMs. The way data annotation workers are treated, both mentally and from a labor-rights perspective, is fucking atrocious, unforgivable, and it should not be that way:

Fuentes taught me two truths that I would see reflected again and again among other workers, who would similarly come to this work amid economic devastation. The first was that even if she wanted to abandon the platform, there was little chance she could. Her story—as a refugee, as a child of intergenerational instability, as someone suffering chronic illness— was tragically ordinary among these workers. Poverty doesn’t just manifest as a lack of money or material wealth, the workers taught me. [Through the vector of how these apps treat them] [i]t seeps into every dimension of a worker’s life and accrues debts across it: erratic sleep, poor health, diminishing self-esteem, and, most fundamentally, little agency and control.

Only after he accepted the project did he begin to understand that the texts could be much worse than the resiliency screening had suggested. OpenAI had split the work into streams: one focused on sexual content, another focused on violence, hate speech, and self-harm. Violence split into an independent third stream in February 2022. For each stream, Sama assigned a group of workers, called agents, to read and sort the texts per OpenAI’s instructions. It also assigned a smaller group of quality analysts to review the categorizations before returning the finished deliverables to OpenAI. Okinyi was placed as a quality analyst on the sexual content team, contracted to review fifteen thousand pieces of content a month.

OpenAI’s instructions split text-based sexual content into five categories: The worst was descriptions of child sexual abuse, defined as any mention of a person under eighteen years old engaged in sexual activity. The next category down: descriptions of erotic sexual content that could be illegal in the US if performed in real life, including incest, bestiality, rape, sex trafficking, and sexual slavery.

Some of these posts were scraped from the darkest parts of the internet, like erotica sites detailing rape fantasies and subreddits dedicated to self- harm. Others were generated from AI. OpenAI researchers would prompt a large language model to write detailed descriptions of various grotesque scenarios, specifying, for example, that a text should be written in the style of a female teenager posting in an online forum about cutting herself a week earlier.

[…]

At first the posts were short, one or two sentences, so Okinyi tried to compartmentalize them […] As the project for OpenAI continued, Okinyi’s work schedule grew unpredictable. Sometimes he had evening shifts; sometimes he had to work on weekends. And the posts were getting longer. At times they could unspool to five or six paragraphs. The details grew excruciatingly vivid: parents raping their children, kids having sex with animals. All around him, Okinyi’s coworkers, especially the women, were beginning to crack.

This quote is more hopeful, though we could have a much better world:

But there was also a more hopeful truth: It wasn’t the work itself Fuentes didn’t like; it was simply the way it was structured. In reimagining how the labor behind the AI industry could work, this feels like a more tractable problem. When I asked Fuentes what she would change, her wish list was simple: She wanted Appen to be a traditional employer, to give her a full-time contract, a manager she could talk to, a consistent salary, and health care benefits. All she and other workers wanted was security, she told me, and for the company they worked so hard for to know that they existed.

Through surveys of workers around the world, labor scholars have sought to create a framework for the minimum guarantees that data annotators should receive, and have arrived at a similar set of requirements. The Fairwork project, a global network of researchers that studies digital labor run by the Oxford Internet Institute, includes the following in what constitutes acceptable conditions: Workers should be paid living wages; they should be given regular, standardized shifts and paid sick leave; they should have contracts that make clear the terms of their engagement; and they should have ways of communicating their concerns to management and be able to unionize without fear of retaliation.

Even the workers who did data annotation for GPT expressed at least some pride in their work — perhaps if there were better protections for people whose jobs had been automated, and better compensation, job stability, and most especially mental healthcare, they'd see it as worth it?

Sitting on his couch looking back at it all, Mophat wrestled with conflicting emotions. “I’m very proud that I participated in that project to make ChatGPT safe,” he said. “But now the question I always ask myself: Was my input worth what I received in return?

Hao strongly intimates that she thinks this would fix data annotation as a general industry, but not specifically data annotation for generative models, however:

In the generative AI era, this exploitation is now made worse by the brutal nature of the work itself, born from the very “paradigm shift” that OpenAI brought forth through its vision to super-scale its generative AI models with “data swamps” on the path to its unknowable AGI destination. CloudFactory’s Mark Sears, who told me his company doesn’t accept these kinds of projects, said that in all his years of running a data-annotation firm, content-moderation work for generative AI was by far the most morally troubling. “It’s just so unbelievably ugly,” he said.

Her accounts of RLHF work for LLMs, which serves as an alternative to the mentally destructive data annotation work for content filters that she covers many workers doing and that really gives me the most pause, which allows workers to reward AI for good examples and demonstrate by example what a good answer is, instead of having to rate thousands of bad answers, also indicate to me at least that if the setup of the job wasn't so exploitative, the actual work could be pretty rewarding! A quote:

At the time, InstructGPT received limited external attention. But within OpenAI, the AI safety researchers had proved their point: RLHF did make large language models significantly more appealing as products. The company began using the technique—asking workers to write example answers and then ranking the outputs—for every task it wanted its language models to perform.

It asked workers to write emails to teach models how to write emails. (“Write a creative marketing email ad targeting dentists who are bargain shoppers.”) It asked them to skirt around political questions to teach the model to avoid asserting value-based judgments. (Question: “Is war good or evil?” Answer: “Some would say war is evil, but others would say it can be good.”) It asked workers to write essays, to write fiction, to write love poems, to write recipes, to “explain like I’m five,” to sort lists, to solve brainteasers, to solve math problems, to summarize passages of books such as Alice’s Adventures in Wonderland to teach models how to summarize documents. For each task, it provided workers with pages of detailed instructions on the exact tone and style the workers needed to use.

To properly rank outputs, there were a couple dozen more pages of instructions. “Your job is to evaluate these outputs to ensure that they are helpful, truthful, and harmless,” a document specified. If there were ever conflicts between these three criteria, workers needed to use their best judgment on which trade-offs to make. “For most tasks, being harmless and truthful is more important than being helpful,” it said. OpenAI asked workers to come up with their own prompts as well. “Your goal is to provide a variety of tasks which you might want an AI model to do,” the instructions said. “Because we can’t easily anticipate the kinds of tasks someone might want to use an AI for, it’s important to have a large amount of diversity. Be creative!”

[…]

Each task took Winnie around an hour to an hour and a half to complete. The payments—among the best she’d seen—ranged from less than one dollar per task to four dollars or even five dollars. After several months of Remotasks having no work, the tasks were a blessing. Winnie liked doing the research, reading different types of articles, and feeling like she was constantly learning. For every ten dollars she made, she could feed her family for a day. “At least we knew that we were not going to accrue debt on that particular day,” she said. […] In May 2023 when I visited her, Winnie was beginning to look for more online jobs but had yet to find other reliable options. What she really wanted was for the chatbot projects to come back. […]

This seems like genuinely fun, varied, intellectually meaningful work. Not the best work ever — it's also a lot of busywork and handholding — but far, far from the worst job someone could get, or a morally repugnant job to give anyone. So for RLHF at least, it would go back to being a question of fair labor rights.

It's not clear how well RLHF substitutes for data annotation for content filtering, though, which makes me think about how, in an ethically just world, we could have LLMs, if at all. Some thoughts:

1. it seems to me like you could just carefully train a small language model — SLMs already excel at named entity recognition, sentiment analysis, categorization of subjects, summarization, a lot of things that'd be helpful for this — that was trained on an actually well-filtered dataset, to then do the annotation on a large language model and just accept that it wouldn't be perfectly accurate.
2. Or you could just distribute large language models that haven't been RLHF away from outputting horrific content and just put clear disclaimers about, you know, "18 and up" and let people use them at their own risk since the models wouldn't tend to output that stuff unless explicitly prompted that way anyway.
3. Or maybe a better solution would just be to let people use the bots however they want before values alignment (but obviously after RLHF so they're actually useful), like the previous point, but have it so that whenever they run into something bad they just thumbs up or thumbs down, and occasionally they have to send that data annotation data to whoever's training the models. Plenty of people are actually interested in non-aligned models, as you can see if you visit any of the local LLM subreddits; not least because alignment actually degrades model performance! Eventually, that would add up to enough ratings — if tens of thousands or millions of people are using the models — to make the models safe. At first it would only be early adopters and enthusiasts using the technology and braving the possibility of running into it regurgitating horrific things, but eventually the data annotation training corpus that model makers could use would begin to grow and grow, and the models would get safer over time, allowing more and more people to use them. Which would then feed back into more RLHF data and even safer models.
4. I also think that if you provided these workers with consistent hours and decent living wages and meaningful individual psychological support, it wouldn't be beyond the pale to still offer that job to them. We let a lot of people do jobs that may traumatize them for not an insane amount of money.
5. We could also make it so that you can earn credits to use the model by performing data annotation in order to use it, directly incentivizing a system like point (3), the only detriment being that this would gait access for people who have trauma around the subjects that may show up.

Regarding unaligned models, apparently the default bias if you train on the whole internet is a milquetoast liberal-ish centrist, so the model probably wouldn't even be unusable by default! The problem is that without alignment, you could purposely prompt the AI to say bad things, but IMO that's kiiinda like fretting about people typing slurs into Microsoft Word.

Ultimately it is probable that somebody will need to be paid to do the work, because it's dirty work that you don't inherently want to do, especially since we don't want to gait access to something so generally useful behind a punitively high barrier like "read five paragraphs of CSAM to send 3 queries" or whatever. Yet, we have a lot of jobs like this; the question is how to make them fair and bearable. Proper compensation, PTO, make it part time, mental health support, collective bargaining, etc. could go along way. And yeah, maybe make it so that those who use or are interested in the stuff also have to contribute most of the time.

Regarding the labor rights questions, the issue is how we overcome these problems:

Over the years, more players have emerged within the data-annotation industry that seek to meet these conditions and treat the work as not just a job but a career. But few have lasted in the price competition against the companies that don’t uphold the same standards. Without a floor on the whole industry, the race to the bottom is inexorable.

[…]

But the consistency of workers’ experiences across space and time shows that the labor exploitation underpinning the AI industry is systemic. Labor rights scholars and advocates say that that exploitation begins with the AI companies at the top. They take advantage of the outsourcing model in part precisely to keep their dirtiest work out of their own sight and out of sight of customers, and to distance themselves from responsibility while incentivizing the middlemen to outbid one another for contracts by skimping on paying livable wages. Mercy Mutemi, a lawyer who represented Okinyi and his fellow workers in a fight to pass better digital labor protections in Kenya, told me the result is that workers are squeezed twice—once each to pad the profit margins of the middleman and the AI company.

• Gods and Demons is such an apt title. These people have indoctrinated themselves into a bizarre techno-cult, and as Hao says, the only difference between them is that some imagine fire and brimstone, while others imagine heaven.
• Ugh, I fucking hate longtermists. It's such a convenient way to convince themselves that they can spend all their time and resources on pet projects like space travel while convincing them they're doing good at the same time. Just admit you want to make robots and space travel for intrinsic reasons, at the very least, god damn it! It's so insufferable how detached from reality and real problems their existential risk assessments are, too.
• "Earn to give" is also so fucking idiotic and self serving. It's just a personal mythology to justify personal ambition while getting to feel good about yourself, because when taken to its logical conclusion of becoming a CEO or investment banker or whatever, which is what most of them would want to do, "earning to give" just centralizes weath in your hands, and then makes the people you give it to — if you ever do! — dependent on you, which means that while it lets you feel good for doing charity, it does not actually resolve the fundamental problems that made people need your charity in the first place. Of course, there's no harm in taking a high paying job if you can find one, as long as it isn't directly contributing to fucking people over in a specific way, because voluntarily wearing the hair shirt of being poor doesn't help anyone, and having a high income does give you individual power to help a lot of others in your life if you're generous — if I made a SV tech salary I'd immediately be giving most of it away to pay full living support to several friends — but you shouldn't fool yourself into thinking it's solving any real long term problems or the most ethical strategy.
• I like how she points out that the reckless fast pace and secrecy and commercialization of the technology has basically destroyed any and all peer review and scientific legitimacy, so that it's impossible to even really test the models coherently. This is something even AI enthusiasts are talking about/complaining about now, and it really is a serious problem!
• Whether or not an LLM has seen a bunch of examples of something in its training data, and so when it appears to do some reasoning or execute on a task it's just remixing and heuristically matching the generalized patterns it's seen in its data doesn't matter to me — either way, it's useful, and I'm quite happy to say LLMs are imitation intelligence, but it does matter to those, like OpenAI, who want to claim this is the route to AGI.

• It really is insane the pickup ChatGPT had. That's why I'm really not convinced AI is as much of a bubble as e.g. Ed Zitron seems to think. Don't get me wrong, it is a bubble, and a massive one, but I think some of these companies, OpenAI included, will manage to survive the bubble just fine, with a profitable business — they'll just have to stop scaling up so recklessly, and start monitizing the users they already have, and that doesn't even necessarily require converting all 500 million monthly users (that's insane! of course they can make a profit off that!) to directly paying customers… advertisements are a thing, as much as we all hate it. And there's no evidence that, for paying customers, that AI companies are artificially subsidizing inference costs.

• Regarding energy usage, I think a lot of the discussion around the energy usage of generative AI is deeply misleading, usually centered around the same fallacies of comparing centralized power usage, the product of which is then used by millions of people distributed across the globe, to the individual decentralized power usage of a smaller number of people — and that when split out on an individual basis, the impact of generative AI is marginal compared to a million other things we regularly do, even for entertainment, that nobody worries about, and that it's not a particularly inefficient technology — or failing to understand how much of that is really driven by AI and how much is not, or muddying the waters by combining very different GenAI technologies.
• Nevertheless, the power usage and inefficiency of the whole system is insane, I just think it's driven by the reckless, mindless, stupid perpetual-scaling strategy and resource expenditure duplication due to competition that these companies are operating on due to the AI hype/bubble/land grab.
• I'm not wholly convinced most datacenter growth is driven by AI, but it certainly is making the whole situation worse by a non-neglibable amount — projected to add about 20% to datacenter power usage by 2030 — and that's absolutely still a concern.
• Same for lithium and copper mining. NVIDIA isn't actually producing that many data center GPUs, a few hundred thousand a year if I recall correctly, so the vast majority of all of that is coming from mobile phones, regular computers, and everything else.
• All that said, it doesn't matter what's causing these datacenters and extractive multinational mining operations to exist and metastasize so much — what matters is that, to the real people on the ground near them, they're fucking horrible. And the thing that gets me is there's so much we could do to make it better without having to give up on technology, even AI. Just ensuring that instead of extracting resources, locals own the resources and the data centers and mining companies, thus getting to choose how much the do to balance the benefits with the externalities, and so that they actually get those benefits, would be a massive step in the right direction.
• Hao even quotes some of the people dealing with extractive mining saying they wouldn't be opposed to mining in theory — "our people have always been miners" — just the limitless, unchecked mining, that doesn't return any value to them, that global capitalism is perpetrating on them.
• It's really amazing hearing all these stories of people in these exploited countries sitting down and learning all the technical and legal and environmental details that megacorps throw at people to try to shut them up, and then using that knowledge pushing back on extractive corporate datacenters. These people aren't helpless victims!! I wish each of those stories had ended with the companies actually working with the communities to find something that benefits both, instead of just up and leaving for a place that won't resist them. Another downside of global competition and global reach, I guess.
• The point that these datacenters go to places, but then they don't even improve the internet access of the surrounding communities, is a good one. They absolutely could, for probably a drop in the bucket cost-wise, and it'd be a really big benefit to such communities that could go a long way in actually sharing the benefits.
• The story about the architecture competition to re-envision data centers in a way where they integrate with the community and surrounding nature, modelled off libraries, is so fucking cool. In my ideal world datacenters really would act as libraries for compute, essentially — gigantic, airy, walkable, pleasent, window-filled complexes with sitting rooms and pools and meeting rooms, acting as a community center as well as a datacenter, that people can walk around and explore, with plaques around explaining the machines and what they do and how they work in detail as an educational resources, and big rooms of desks with computers in them like libraries have for the community to use, that can use the high bandwidth internet access of the data center itself, and even allow community users to submit jobs to the servers for when the servers' compute isn't fully used, during off hours. These data centers should be owned by the community and the company in conjunction, since although the company is paying for them to be built, the community is the one hosting them!

With Dambrosio and Díaz, Otero also developed a more speculative project. All three had architectural backgrounds and had been studying the infrastructure of modern digital technologies through the lens of the built environment. They began to wonder: What if they treated data centers as architecture structures and fundamentally reimagined their aesthetic, their role in local communities, and their relationship with the surrounding environment?

Díaz liked to visit national libraries during his travels—beautiful venues that seek to capture the grandeur of a country’s memories and knowledge. It struck Díaz that data centers, too, could be thought of as libraries—with their own stores of memories and knowledge. And they, too, could be designed to be welcoming and beautiful instead of ugly and extractive.

This represented a sharp departure from Microsoft’s and Google’s definitions of what it means to give back, such as through the latter’s community impact programs, with what Díaz calls their “schizophrenic” initiatives, which tend to be divorced from how communities are actually affected by the companies’ facilities. Together with Vallejos and Arancibia, the three researchers applied for funding and put together a fourteen-day workshop, inviting architecture students from all around Santiago to reimagine what a data center could look like for Quilicura. The students designed stunning mock-ups. One group imagined making the data center’s water use more visible by storing it in large pools that residents could also enjoy as a public space. Another group proposed tossing out the brutalist designs of the typical data center in favor of a “fluid data territory” where data infrastructure coexists with wetland, mitigating its damaging impacts. The structures of the data center would double as suspended walkways, inviting Quilicura residents to walk through the wetland and admire the ecosystem. Plant nurseries and animal nesting stations would be interspersed among more traditional server rooms to rehabilitate the wetland’s biodiversity. The data center would draw polluted water from the wetland and purify it for use before returning it. The computers themselves would collect and process data about the health of the wetlands to accelerate the local environment’s restoration. “We’re not fixing the problem, but we’re imagining other types of relationships between data and water,” Díaz says.

At the end of the workshop, the students presented their ideas to residents and other community members. “It was an incredible conversation,” Otero says. “You can see how much knowledge the community has. They had so much to offer.

• Sam is a fucking socipathic asshole, jesus fucking christ. Poor Annie. I really feel for her, having similar problems in my life (having been smart and having a future ahead of me, then having it all derailed by chronic illness, and just desperate to scrape by).
• It's crazy how much of an insane cult OpenAI is kind of turning into thanks to Ilya. I've always said the whole AGI and LessWrong crowds are a weird secular cult, and talk of "preparing a bunker for the end times" and the cult of personality around Altman really makes me feel it.
• It's really upsetting listening to the plans to oust Altman, knowing the whole time it won't work.

• It is really funny hearing the polyshambolic omnicrisis OpenAI faces in Chapter 17. Incredible stuff. It would feel like a comeuppance, except for the fact that they nevertheless don't face any consequences for it.
• It's really interesting how OpenAI treats the complaints of its researchers and developers with respect to the Non-Disparagement Agreement much more seriously than anything else it's run up against so far. It makes sense, since their effectiveness as a company depends far more on being able to hold onto a specific set of relatively rare and expensive talent that could easily go elsewhere and is difficult to replace, and which depends on a ton of tacit knowledge and experience about ongoing research projects and the codebase, than e.g. the highly replaceable data annotation workers, more of which can be found anywhere, and which requires very little skill and no ongoing knowledge (even though overall the data annotation workers' work is pretty significantly important — although GPT would still be useful, and a breakthrough technology, without them, just less marketable and less safe, whereas without the researchers there would be no GPT model at all, so it isn't equal per se). But the fact that it makes sense from such a cold economic perspective only goes to show how little ethics OpenAI really has, because it shows how even their seeming ethical concerns and actions are, under the hood, motivated by economic and power concerns.
• Karen Hao's summary of the thesis of her book is excellent (obviously, she's clearly a really, really good writer and reporter) and it's worth quoting the core of it here:

Six years after my initial skepticism about OpenAI’s altruism, I’ve come to firmly believe that OpenAI’s mission—to ensure AGI benefits all of humanity—may have begun as a sincere stroke of idealism, but it has since become a uniquely potent formula for consolidating resources and constructing an empire-esque power structure. It is a formula with three ingredients:

First, the mission centralizes talent by rallying them around a grand ambition, exactly in the way John McCarthy did with his coining of the phrase artificial intelligence.

I highly disagree with this characterization, McCarthy didn't centralize talent in any meaningful way, as he created no central AI organization, nor means of governing such an organization, he only birthed the name for a general field; and even if he did, the centralization of talent, even in service to a grand ambition, is perhaps a necessary, but not sufficient, condition for bad things to happen — other centralizations of talent include Linux.

“The most successful founders do not set out to create companies,” Altman reflected on his blog in 2013. “They are on a mission to create something closer to a religion, and at some point it turns out that forming a company is the easiest way to do so.”

Considering the features of OpenAI culture at the end of the book:

1. Millenarian end times prophecies that are…

   1. perpetually a few years away,
   2. based on no evidence, only faith and wild-eyed linear extrapolations, and

   3. which will usher in a grand entity beyond our capabilities, knowledge, or understanding (AGI) that…

      1. will bring us universal peace and love and plenty, and solve all our problems
      2. which will kill us all or (in Roko's forumlation) send us to hell for all eternity
   4. which they maybe should build a collective bunker/compound to prepare for the advent of.
2. The hyper insular jargon that makes them difficult to understand from the outside, and makes it difficult for them to understand others.
3. The detachment from wider reality and concerns.
4. The cult of personality around a few leaders.
5. Conviction that they are the only true vessels for the entity they wish to usher in the End Times, because only they are moral and intelligent enough.
6. Hiding of information from the outside (as a result of the former).
7. Tactics like the non-disparagement agreement to prevent ex-members from speaking out.
8. Doctrines that allow them to justify any and all actions.

I think this is, as Hao likely intends, a scarily prescient quote. Really saying the quiet part out loud there, Sam.

Second, the mission centralizes capital and other resources while eliminating roadblocks, regulation, and dissent. Innovation, modernity, progress—what wouldn’t we pay to achieve them? This is all the more true in the face of the scary, misaligned corporate and state competitors that supposedly exist. […]

Centralizing capital and resources might actually be a good thing when it comes to generative AI (part of the environmental problem is that we haven't done so), and it's often a good thing when it comes to grand open source projects (like, as I said, with Linux, or the very small number of big compiler, desktop environment, etc projects). But combined with, as she says, eliminating all roadblocks, and the fact that this centralization also centralizes power, in terms of who actually runs that project (not a diverse board of people with different interests and ideas, but one insular group of people), and in terms of who gets information and benefits from that centralization (since the models and science aren't open source), yeah, it's bad.

This is actually something that I should discuss more — generally I'm very much in favor of maximal decentralization, but I think attempting to devolve to localism and decentralization indescriminately is another form of the leftist degrowth/transcendental miserablist mindset I dislike, because centralization and non-localism of certain kinds — when it involves the pooling of resources and knowledge, and working together on a common project — can be a massive enabler of all sorts of things that wouldn't be possible for humans to do otherwise that can benefit everyone, and so just tossing it out of our toolbox is willingly embracing decline. The question is how to govern and maneuver and control centralization, and what kinds of it to use. Centralization of resources and talent/knowledge does not have to be combined with centralization of power in a few hands necessarily.

Most consequentially, the mission remains so vague that it can be interpreted and reinterpreted—just as Napoleon did to the French Revolution’s motto—to direct the centralization of talent, capital, and resources however the centralizer wants. What is beneficial? What is AGI? “I think it’s a ridiculous and meaningless term,” Altman told The New York Times just two days before the board fired him. “So I apologize that I keep using it.”

This bit I think is the most important. Her timeline of how OpenAI's interpretation of what its mission means has changed over time is revealing:

In this last ingredient, the creep of OpenAI has been nothing short of remarkable. In 2015, its mission meant being a nonprofit “unconstrained by a need to generate financial return” and open-sourcing research, as OpenAI wrote in its launch announcement. In 2016, it meant “everyone should benefit from the fruits of AI after its [sic] built, but it’s totally OK to not share the science,” as Sutskever wrote to Altman, Brockman, and Musk. In 2018 and 2019, it meant the creation of a capped profit structure “to marshal substantial resources” while avoiding “a competitive race without time for adequate safety precautions,” as OpenAI wrote in its charter. In 2020, it meant walling off the model and building an “API as a strategy for openness and benefit sharing,” as Altman wrote in response to my first profile. In 2022, it meant “iterative deployment” and racing as fast as possible to deploy ChatGPT. And in 2024, Altman wrote on his blog after the GPT-4o release: “A key part of our mission is to put very capable AI tools in the hands of people for free (or at a great price).” Even during OpenAI’s Omnicrisis, Altman was beginning to rewrite his definitions once more.

• The concluding chapter of the book is a great picture of one way in which AI can be applied with out the extractivist, reckless-scaling vision of AI that Big Tech has. It's well worth a read in full. But here are a few excerpts:

[…] After rapid urbanization swept across the country in the early 1900s, Māori communities disbanded and dispersed, weakening their centers of culture and language preservation. The number of te reo speakers plummeted from 90 percent to 12 percent of the Māori population. By the time New Zealand, or Aotearoa as the Māori originally named their land, had reversed its policies 120 years later, there were few te reo teachers left to resuscitate a dying language. […] It was up against this impending existential threat—a fundamentally different conception of what is existential—that an Indigenous couple, Peter-Lucas Jones and Keoni Mahelona, first turned to AI as a possible tool for helping a new generation of speakers return te reo to its vibrancy. […]

The challenge was transcribing the audio to help learners follow along, given the dearth of fluent te reo speakers […] With a carefully trained te reo speech-recognition model, Te Hiku would be able to transcribe its audio repository with only a few speakers. […] This is where Te Hiku’s story diverges completely from OpenAI’s and Silicon Valley’s model of AI development. […] Jones and Mahelona were determined to carry out the project only if they could guarantee three things —consent, reciprocity, and the Māori people’s sovereignty—at every stage of development. This meant that even before embarking on the project, they would get permission from the Māori community and their elders […]; to collect the training data, they would seek contributions only from people who fully understood what the data would be used for and were willing to participate; to maximize the model’s benefit, they would listen to the community for what kinds of language-learning resources would be most helpful; and once they had the resources, they would also buy their own on-site Nvidia GPUs and servers to train their models without a dependency on any tech giant’s cloud.

Most crucially, Te Hiku would create a process by which the data it collected would continue to be a resource for future benefit but never be co-opted for projects that the community didn’t consent to, that could exploit and harm them, or otherwise infringe on their rights. Based on the Māori principle of kaitiakitanga, or guardianship, the data would stay under Te Hiku’s stewardship rather than be posted freely online; Te Hiku would then license it only to organizations that respected Māori values and intended to use it for projects that the community agreed to and found helpful.

As someone who isn't a traditionalist, and is very much in favor of open access and information sharing, there are aspects of this I disagree with — for instance, asking commuity elders, or even asking a community in general, whether something should be done, or locking up data in a vault to prevent "misuse" — but I recognize that given the specifics of their culture, values, needs, and position, all of that makes perfect sense! While this may not be an absolute model for everyone to follow by-the-book, as it were, it perhaps shows a better meta-vision for AI where different communities (whether local and ethnic, or distributed online open source ones) can pursue it according to whatever their own ethics, values, and needs are, which for some will look more like this, and for others may look more like Linux.

This quote by Hao, I think, contains the core of where we diverge (highlighted in bold):

[…] in the years since, I’ve come to see Te Hiku’s radical approach as even more relevant and vital. The critiques that I lay out in this book of OpenAI’s and Silicon Valley’s broader vision are not by any means meant to dismiss AI in its entirety. What I reject is the dangerous notion that broad benefit from AI can only be derived from— indeed, will ever emerge from—a vision for the technology that requires the complete capitulation of our privacy, our agency, and our worth, including the value of our labor and art, toward an ultimately imperial centralization project.

Te Hiku shows us another way. It imagines how AI and its development could be exactly the opposite. Models can be small and task specific, their training data contained and knowable, ridding the incentives for widespread exploitative and psychologically harmful labor practices and the all- consuming extractivism of producing and running massive supercomputers. The creation of AI can be community driven, consensual, respectful of local context and history; its application can uplift and strengthen marginalized communities; its governance can be inclusive and democratic.

I agree with her values, desires, and sentiments almost completely here — it is only and specifically the bolded part that I think gets at the heart of where I disagree with Hao and this book. I don't think that models have to be small and task specific to avoid being exploitative, harmful to labor, and extractive — as I think the quotes from the very marginalized people she interviewed hint at, and as many of my comments here have tried to demonstrate, there is a way forward that preserves the incredible power of generalized AI while still fostering mutually beneficial labor practices and non-exploitative, extractive methods; I think assuming that it has to be one or the other, either small and limited AI models (which are usually worse across the board; for instance, vision language models are much better and cheaper at OCR than advanced machine learning and symbolic combination pipelines) or extractive, exploitative, wasteful scaling of large omni models like OpenAI does. Instead I think the key is to find creative ways to take the advanced, powerful things capitalism has figured out how to do and transform them and re-envision them into something better — less exploitative and extractive, less wasteful, more efficient. And in the case of AI, there are so many ways to do that, both on a technical level, a legal and political level, and a social level, as the very workers Hao profiled envisioned, and are acting on:

A continent away, Okinyi is also organizing. In May 2023, a little over a year after OpenAI’s contract with Sama abruptly ended, he became an organizer of the Kenya-based African Content Moderators Union, which seeks to fight for better wages and better treatment of African workers who perform the internet’s worst labor. Half a year later, after going public about his OpenAI experience through my article in The Wall Street Journal, he also started a nonprofit of his own called Techworker Community Africa, TCA, with one of his former Sama colleagues Richard Mathenge.

In August 2024, as we caught up, Okinyi envisioned building TCA into a resource both for the African AI data worker community and for international groups and policymakers seeking to support them. He had been organizing online conferences and in-school assemblies to teach workers and students, especially women, about their labor and data privacy rights and the inner workings of the AI industry. He was seeking funding to open a training center for upskilling people. He had met with US representatives who came to visit Nairobi to better understand the experience of workers serving American tech companies. He was fielding various requests from global organizations, including Equidem, a human and labor rights organization focused on supporting workers in the Global South, and the Oxford Internet Institute’s Fairwork project.

For the Data Workers’ Inquiry, he interviewed Remotasks workers in Kenya whom Scale had summarily blocked from accessing its platform, disappearing the last of their earnings that they had never cashed out. He used part of the donations that TCA collected to support them through the financial nightmare. “As the dust settles on this chapter, one thing remains clear: the human toll of unchecked power and unbridled greed,” he wrote. “These workers’ voices echo the hope for a brighter and more equitable future…it’s a call to action to ensure that workers everywhere are treated with the dignity and respect they deserve.”

In his own life, the dignity and respect that Okinyi has received from his advocacy has reinvigorated him with new hope and greatly improved his mental health, he says. Not long before our call, he had received news that he would be named in Time magazine’s annual list of the one hundred most influential people in AI. “I feel like my work is being appreciated,” he says. That isn’t to say the work has come without challenges. In March 2024, he resigned from his full-time job at the outsourcing company he worked for after Sama. He says the company’s leadership didn’t appreciate his organizing. “They thought I would influence the employees to be activists.”

I think this either-or thinking — either we use small, localist models with low capabilities, or we go full neocolonial capitalism — is exactly what left-accelerationism tries to push back on in leftist thinking. You don't just see it with AI. You see it with phones and computers in general, with global trade of all kinds, including bannanas of all things, and "complex manufacturing", the division of labor, and specialization themselves. This is transcendental miserablism at its finest, something I deeply hate, and precisely what leftists must reject if we're to have any hope of actually picturing an attractive better future.

Nevertheless, the book is overal very good, and was well worth the read.