Category: Philosophy

When using a large language model to gain knowledge or perform tasks, they remind me of the Library of Babel: they're capable of outputting basically all grammatical assemblages of tokens, and their probability distribution contains (a fuzzily associative, highly compressed, copy of) essentially all of human knowledge and thought. Thus contained within it is the complete catalogue of useful, insightful, correct, and wise things a human being might say, and all the wrong, dumb, or plain nonsensical things a human might say, just as the Library contains all of that and also all of the nonsensical assemblages of letters around it. Also like the Library, even if you find something within the total distribution that appears coherent, you don't know if it's correct — after all, the library contains "the faithful catalogues of the Library, thousands and thousands of false catalogues, the demonstration of the fallacy of those catalogues, the demonstration of the fallacy of the true catalogue" — so you can only gain something useful from the library if either, one, you already know the answer, two, don't need the answer to be correct, or three, have a way to verify the answer to the degree you need it to be correct.

Thus the boundless hope and optimism with which some view large language models, because useful combinations of tokens can be found within that distribution, and likewise the endless depression others find therein, because for every useful output there is a useless one as well, or ten, or a hundred:

When it was proclaimed that the Library contained all books, the first impression was one of extravagant happiness. All men felt themselves to be the masters of an intact and secret treasure. There was no personal or world problem whose eloquent solution did not exist in some hexagon […] As was natural, this inordinate hope was followed by an excessive depression. The certitude that some shelf in some hexagon held precious books and that these precious books were inaccessible, seemed almost intolerable.

The differences between LLMs and the Library are:

1. that not every output is equally probable: only generally grammatically correct and seemingly-comprensible ones, usually;
2. you can direct an LLM to different parts of the probability distribution, guiding them and increasing your chances that the Library will produce the knowledge you need;
3. you can control how likely or unlikely the outputs they generate are, to expand and contract the space within which you must search.

Thus, as long as the conditions for the usefulness of a Library hold, this one is much more useful than Borges'. Nevertheless, many of the same psychological and almost theological phenomena result: some people view this fact, that we've compressed essentially all of knowledge and human thoughts patterns into 500 gigabytes that you can direct to a space of probability within that wider probability curve with awe and curiosity, and see the possibilities; others are filled only with despair that much of what this thing will produce is bullshit.

When using LLMs as conversational mirrors, however, LLMs remind me of something stranger. In Nova, Samuel R. Delany argues that the Tarot is in fact a useful tool for cognition:

Mouse, the cards don’t actually predict anything. They simply propagate an educated commentary on present situations…The seventy-eight cards of the Tarot present symbols and mythological images that have recurred and reverberated through forty-five centuries of human history. Someone who understands these symbols can construct a dialogue about a given situation. There’s nothing superstitious about it. The Book of Changes, even Chaldean Astrology only become superstitious when they are abused, employed to direct rather than to guide and suggest.

As this article by someone who actually practices such a thing expands:

A blind faith in the tarot as a means of predicting the future, like any form of divination, is characteristic of childhood, and most of us mature to the point where we no longer take it seriously. For artists like Delany, however, there’s another stage beyond this, in which we realize that structured repositories of arbitrary symbols—often preserved in the cultural memory as oracles—can be a tool for thinking through questions that are resistant to more rational analysis. Writing about the tarot a few years ago, I called it "a portable machine for generating patterns," and I noted that it results in a temporary structure, spread out across the table, that feels like an externalization of the problem in your head. You can examine it, scrutinize it from different angles, or even walk away from it. […] It won’t tell you the future, but by forcing you to map or analogize your current situation onto a matrix of charged symbols, it can provide surprising insights into the present.

I propose that one way to view large language models, is as something like this: talking to an ungrounded large language model (e.g. having it play devil's advocate and ask you questions and try to disprove you and stuff like my dad does) functions as sort of like doing a tarot reading. It cannot predict the future, or even accurately describe the present. There is no truth or mind or meaning there inherently — it's all assigned by your mind. But through the act of trying to map whatever discourse or concern you have onto this semi-randomly generated pattern, you may get a new perspective or be jogged out of limited ways of thinking. And through the act of trying to analogize the entities operating in whatever discourse you're concerned with, to the layered meaning of the symbols on the table that it is manipulating uncomprehendingly, you may layer more interesting meaning or uncover more of your own personal unconscious thoughts. As long as you remember that you can always dismiss what the cards, or the model, have so positively said as meaningless or useless and walk away from the table, you may still glean something useful from it.

I've run a few experiments essentially using LLMs this way and it seemed profitable — taught me a few things about myself I hadn't phrased explicitly before, through asking me randomly generated questions. I think when it comes to the Tarot itself, his argument is somewhat flawed, at least as it applies to me, but with LLMs it is perhaps less so, and works better for my more literal, less mystical nature, since the patterns it creates are not purely random, but based on a probability space trained on all human reasoning and thought patterns, and since the symbols it works with are all of human language, not just a few set archetypes. It mimics the patterns it has seen of criticism, critique, questioning, and sometimes it's able to ask insightful questions that challenge me to construct new answers, whereas other times it may ask nonsense or say things that are not well reasoned, which I can ignore. And unlike another human interlocutor, I can dump entire pages from this blog, collections of writings on various topics that can sometimes reach into the tens of thousands of words, into it, and it can combine that with reading any of the texts I link to, and all of its compressed world-knowledge, to find the right patterns to question me about, thus giving it infinitely more specificity and context than any human interlocutor I'm likely to find, as well as infinitely more patience.

Is it a replacement for discussion with a human being, who can more trenchantly challenge my views? No. It tends to accept, or at least not challenge, the underlying moral values and assumptions I make, which is nice when I don't want to stand around arguing with a nazi, a statist, a Luddite, or a primitivist, but sometimes it's worth arguing with those — well, not the first one — to sharpen my wits. But you can do both/and.

Basically, the central premise of the series is that there was this gigantic war between hyper-conservative fundamentalist religious and fascist factions back in the day. They call it the "Church Wars", but it’s unclear exactly how much it was strictly religion per se, and how much of that perception is retroactive propaganda on the part of the current society, in light of the attitude indicated by their reaction to it — namely, to completely ban all public expression of gender, gender roles, and beliefs regarding religion, metaphysics, or philosophy, outside super-basic ethics and politics. However, this was done only about 250 years ago at the time the novels take place, and it wasn’t as if the culture was really leading up to it when it was done; it was just a massive, panicked overreaction to the nature of the prior war.

And so their culture still has all of these ideas about gender and religion and so on floating around—picked up from the subconscious and semi-conscious ways people are treated differently, from cultural histories and traditions, from works of art left over from previous generations, etc. The only thing that’s changed is the surface-level ability to consciously acknowledge and discuss these things and call them by their true names has been tamped down, and thus, as a corollary, to even point out their influence!

More than that, nobody has figured out yet how to express something to replace the very powerful, rich, interwoven, well developed social constructs of gender (and communal, shared religious, philosophical, and metaphysical discussion, debate, and belief together). They seem to have gone with the stereotypical leftist conception of gender-neutral, meaning a riotous and incongruous conglomeration of various signifiers which, ripped from their context and juxtaposed, mean nothing, signal nothing, refer to nothing, and impart nothing, and thus fades into ugliness or noise; or a lack of all signifiers at all in a sort of beige unisexuality which is more the lack of expression and interest than anything else. Nobody’s built any interesting or meaningful social roles or ideas or ethics or modes of behavior or anything! They haven’t built anything out of it. It’s just sort of gender acceleration: a riotous nothingness that leaves no one with anything to latch onto and no roles to inhabit. Thus people are left vulnerable to — because they still recognize — and hungry for — because of the lack that hasn't been provided for, because even recognizing there is a lack would be inconceivable — Old Things.

In the midst of all this, a very well-educated combination priest and psychologist establishes, essentially, a secret brothel-slash-religious and political discussion parlor in the style of the various parlors during the French Enlightenment that were a huge influence on the rich and powerful in that time: inviting people in to experience all of these taboo topics around sex and gender and religion and metaphysics and so on, to discuss them and debauch in them and revel in them without judgment and with perfect secrecy. Then they’d meet up and network after and have discussions about politics and build connections.

The whole concept of the parlor/brothel is that it’s essentially like a really hardcore 1700s role-playing time-vortex, with some of the Madame’s children actually having been raised in that environment and ideology from birth.

This obviously attracts a lot of the powerful and kind of traps them in her orbit first by the intrigue of violating these taboos and also getting to secretly network and talk to each other and conspire, and then later by the threat of blackmail that she can hold over them.

The main narrator has kind of been adopted by the Madame and the rich and powerful in her orbit, and so obviously his ideology and perceptions of the world are very much informed by hers. So of course he has an obsession with French Enlightenment philosophers and ideas, which he weaves into the narrative and of course structures the narrative and even the prose very much in the style of. But he’s also convinced—and perhaps not necessarily wrongly—that although race and religion and gender and so on have been nominally eliminated from the public sphere (race because unrelated technological advancements have essentially eliminated the geographic nation and made traveling around the world take about four hours. So there’s been so much mixing of races and so much detaching of race from anything but culture that people don’t think about it anymore), their implicit presence is very important to understanding how the elites that cause the war that this series covers think, because of the Madame's influence, but also in general for understanding all of the interactions and things that are going on, since its form is informing people’s unconscious biases.

So he’ll assign he/him and she/her pronouns to people based on what role he thinks they’re inhabiting in the moment—even at the same time as he recognizes their sex characteristics (which may not align)—because, as he insists, those two are noticed by people and influence how we treat people based on our attraction and level of physical imposingness and so on, even if we don’t like to admit it. This, of course, is kind of silly because it then narrows gender: so strong and dominant people are all he. And weak or caring or manipulative people are all she. And he protests much too strongly against the idea that he’s doing that, but I think he definitely falls into that really hard.

On top of that, there’s the fact that things do that transportation system and the end of the geographic nation. What has replaced it is the institution of the hive, which is a voluntary government that you can give your allegiance to no matter where you are or who’s around you or what territory you’re in or whatever. That then gives you the laws that you’re bound by and will also stand up for you in dispute resolution with people from other hives and resolves disputes between you and people at the same hive.

These hives are all based around all sorts of different ethos, and at the age of majority you choose one to be a part of. What this has led to is a sort of self-selection where each hive is really very concentrated in terms of having the same people with the same general ethos as in it, leading to various hives having monopolies over various key social functions and turning the hives into an almost gender system of its own, but one which is not allowed to be related to sex and romance and hasn’t really been developed or elaborated because of the way people are trying to tamp down on a gender system.

(There are also hive lists who are not governed by any laws. They seem to be doing pretty well to be honest and have a seat in the global congress.)

This all has led to a sort of resurrection of things like dictatorship. Because if you can pick and choose who’s laws you submit to, then having one sort of law system be a dictatorship isn’t that much of a problem because if the dictator goes crazy or makes bad choices, you can just switch. In addition, the need to woo members through an ethos and a vision leads to that being a pretty promising option.

At the same time, the Democratic hives have fallen afoul of demagoguery and tend to elect single central powerful leaders. And the one hive that tries a truly new government system, one based on bureaucratic technocracy where everyone submits suggestion letters, and then those are sorted and summarized by a machine, has led to a situation where whoever runs the sorting algorithm basically runs the government.

So everything is very centralized. Especially since you need hives to be a certain size to be able to stand up to the other ones meaningfully, so there is a sort of growth tendency. That’s why there are only six hives.

This leads to a narrative where obviously a lot of the big powerful politicians and change makers are the ones who really matter to the development of the global political scene.

Oh also—the "great men" in this story are often hilariously stupid, petty, venal, horny, just like all nobility. That’s half the fucking point. And their power, for the most part, is an obvious failure of the system the novel clearly isn’t endorsing. That they have redeeming qualities too—especially when seen through the narrator’s too-kind eyes—is only the mark of a writer who isn’t so blinded by morality they don’t understand the appeal of antiquity. Because nobility does have appeal. That’s like… how we had it for as long as we did.

So, of course, basically all of the reviews are just constantly complaining about Great Man Theory (as if powerful individuals can't possibly be acknowledged to have large effects on history or else the book is bad) and how sexist the book is (because the narrator is), and most of all how "regressive" it is for recognizing that there is an allure to gender and dictatorship and empire and all of these things!

Because how dare Ada Palmer, right? How dare she explore the fact that, yes, these things are attractive to the human soul — we are not angelic beings for living in the 21st century (or the 25th!), we have not "evolved" past the instincts and needs that make things like hierarchy and social roles attractive to us, nor past the cultural associations that make the 18th century seem alluring to some degree. How dare she explore and understand that tough subject, when understanding precisely those hungers, those needs, is what would allow us to beat back actual reactionary danger? You can't win by just offering people nothing in return for what they lose but moralistic secular sermons.

Rolling back to gender for a second, one of the things that bothers me a little about the idea of being nonbinary is that it's really just defining itself in opposition to the two contentful gender categories we have; it has no actual content in itself. So conservatives are a (very) little right when they ask "what the fuck is nonbinary?" because ultimately, it kind of isn't anything half the time, it's just a sort of rejection of other things without constructing anything meaningful in its place, while also trying to hold onto gender in some way, unlike agender people, so it's just existing in this meaningless beige inbetween zone with a riot of signifiers and pieces but nothing that means anything.

And to be sure, I've seen my share of people who actually have a good answer to this question, but even then, a lot of it is hyper-individualized, and the problem with that is that gender, when you boil it down, is literally just a genre of person, not the person itself. So for this concept of gender to have any sort of meaning or utility, we need some sort of protocol agreed on by some group.

When people boil down gender to just… 'how I feel' then, well, I'm not going to be the Gender Police and tell them to "get a real gender." I'm not even really going to look them askance, because I get it. But I always have a small sense of them having missed the point. Gender is an identification-with, but at that point they've reduced the concept of gender to be no different from their unique self, which then puts us in an awkward position because the whole point of the concept is it provides an abstraction – some minimal set of things that represent what kind of person you are so that people can treat you accordingly. Obviously you are more than your gender, but in programming speak, narrowing someone's type from Unique#71837 to Girl helps us a lot.

If someone is a Girl you have methods and fields you know about and can call that won't throw exceptions. Obviously it's never perfect but it does minimize the number of exceptions we have to catch and do domething about per person — and if there are too many exceptions, well then, it's time to create a new gender the way you'd create a new class/type. Ultimately, I'd want genders to become what some music scenes or things like butch-femme are: an infinitely diverse and kaleidescopic array of actual subcultures with real histories and ideas and expectations built into them to choose from. A sort of acceleration of gender production that stops short of the g/acc nihilism of one-gender-per-individual thus leading to the extinction of gender in toto.

You can have an individual interpretation of your gender too – stuff like what you relate to and see as relevant for the kind of person you are in society, but in terms of anything relevant to how others treat you, yeah, agreement and conversations have to happen to sync everyone's models up at least somewhat.

Also when I say a lot of nonbinary gender expression and conceptualization is beige, I don’t just mean the unisex theyfab stuff. I also mean like the stuff that’s a total riot of signifiers that all contradict each other and, ripped from their context and put together, mean absolutely nothing. Pure entropy also looks like a smooth beige from far enough away.

The point is not to police people into A Few Gendered Categories, but to say that genders, as The Major says, should be abstractions that bring with them protocols for understanding and interacting with you. You can’t have that if they’re completely unique to each individual. Gender is genre.

And, since I was at work, I didn’t get to express this as strongly—I was mostly listening to her and nodding along. I think it’s important to understand that gender receives most of its meaning and function from common identification with others, from becoming part of a community when you adopt it, and seeing others like you and defining yourself with respect to them—so if you’re the only one that holds your unique gender, you can’t do that, and it just becomes, as you say, expression or personal identity. Not gender.

I recently read something that made me deeply sad and angry:

According to The Harvard Gazette, Becker said that as a child, he was a firm believer that humanity should live among the stars but his belief changed as he learned about the inhospitality of space. "As I got older, I learned more and realized, 'Oh, that's not happening. We're not going to go to space and certainly going to space is not going to make things better," he said. — "The Stupedist Thing": Scientist Shreds Elon Musk's Mars Dream

I hate leftists. Yes, Mars as a lifeboat is a dumb idea. But pooh-poohing the idea of going to space like this is just gross. It's transcendental miserablism of the worst kind: a total abandonment of grander aspirations for the future, of dreaming and love and romanticism, in favor of resolutely keeping our noses in the dirt and shouting down any hopes for anything else, any striving for other things, even if that striving has historically been beneficial to the present.

It's reminiscent of how Utopia is treated in Terra Ignota. In the books, there's a Hive called Utopia whose members all pledge their lives to defeating death one cause at a time, to getting humanity to the stars, to enhancing humanity, and in every way bringing wonders to life. They know it will be hundreds of years yet, if it's even possible, to achieve the things they aim for, but they see the struggle as worth it, and they sustain themselves with a love of stories and beauty and whimsey enabled by a love of beautiful and amazing technolog. They work tirelessly to get there… and everyone hates them for not focusing on the here and now, on Earth.

I really hate that leftism seems to have given up that hope, so much so that now they just have a slur for it ("techno-optimist"). I agree that the actual intended usage of the term refers to something I disagree with and think is wrong — tech won't automatically make everything better! — but nowadays the word seems mostly to be used against people who even think that technological progress could make things better, and that it's worth trying to learn and grow and expand science and technology, to see what grand things we can do, even if we don't have an immediate story for the payoff of those research projects, or even if we haven't perfectly predicted all the consequences those projects might have. It's sad.

In my opinion, they have fundamental flaws that mean that you can't use them for many of the things people are claiming you can use them for, such as obtaining factual information, programming, or writing things for you. This becomes clear if if you look at how large language models actually work:

• they are not oriented towards producing true statements in the way that an honest human can be (although this is to a degree changing thanks to reinforcement learning using accuracy rewards), and can be substantially mitigated, to the point where they're still useful for knowledge gathering, through tools that use resource-augmented generation and internet search to provide them with sources that they summarize and synthesize and then cite, which, critically, the user can then go look at (see: Open-WebUI's Web Search feature, Perplexity.AI, and Perplexica for, imo, the best implementations of this)
• their hallucinations are mathematically ineradicable
• they are incapable of double checking their work, meta-cognition, applying rules to their thought processes, and so on, because their fundamental structure is only feed-forward
• their emergent capabilities are probably an artifact of noncontinuous scoring criteria
• they don't have reliable or particularly good actual abstract reasoning or understanding capabilities, but mostly rely on applying more rote patterns they've seen from their training sets because learning to reason from data is fundamentally impossible
• claims of them reaching human-comparable scores on tests come from deeply flawed, borderline fraudulent practices (another much more recent example)
• they are clearly just an entropic process palely regurgitating degraded forms of previous information, not producing new information, because they devolve when trained on their own outputs
• they are incapable of producing actually good poetry or prose, or anything inherently worth reading by itself, because their ideas and ways of expressing them are banal by nature (they generate the most likely string of text, so it will always be middle of the road, even if you input an unusual idea it'll just regress to the mean)
• they cannot be secured
• they generate less maintainable code with more common security vulnerabilities, and can't reliably independently perform even basic coding tasks, even if developers produce code "faster" using them
• The fundamental model of large language models is incompatible with most important programming tasks: when you're actually programming, small mistakes or inaccuracies and general bullshit in code that by definition is designed to look convincing can be a serious problem, because they'll be hard to spot – especially since you'll be less familiar with code you didn't write every token of yourself, and writing code is inherently easier than reading or debugging it
• The studies that seem to indicate large language models improve coding performance are full of errors, holes, and small sample sizes.
• One particularly good study shows that usage of AI probably slows experienced developers on large software projects down.
• etc.

Furthermore, I think the hype around large language models is extremely harmful. Not only because it's creating an entire sphere of grifting that's taking advantage of people, and inflating a market bubble that will inevitably pop costing many people their livelihoods in the process, but also because while we shouldn't be afraid that large language models actually can take the jobs of writers, journalists, and programmers, the fact that they are being marketed as capable of doing so will be equally damaging. Capitalists have an inherent strong incentive to want to decrease labor costs and deskill and disempower workers, and the marketing hype that convinces them that we can be replaced with LLMs will enable them to do that whether or not LLMs can actually replace us.

On top of all of that, there's the way that the use of large language models for things like programming – and even writing – can effects learning. If instead of actually going through the trouble to actually understand how things work and why at your chosen level of abstraction in the software stack, or even delving below it so that you can use your abstractions more effectively (since all abstractions are leaky) – the principles, rules, and relations between things – and more importantly building up the meta-skills of problem-solving, critical-thinking, and autodidacticism, you instead rely on the AI to do your thinking for you, it can have serious consequences for your intellectual development.

That is what the study that is often misquoted by AI Luddites actually says. Not that having any contact with AI magically "rots your brain" or makes you "permanently dumber", as many headlines have made it seem, but the more limited and specific claim that using AI to automate actually performing a task, including reading, processing, and then synthesizing information, leads you to know less about the subject and how that task was executed — an extremely obvious outcome from the basic concept of automation, and not something to panic over, just to be aware of: that if you automate something, you know less about how it was done and in what way.

This would theoretically be fine if the AI was deterministic, so you could actually rely on it to behave in a reliable and understandable way, and if it didn't make mistakes or made mistakes in consistent and comprehensible areas, like a compiler almost, but AIs are the leakiest of all possible abstractions over real code, which means when something goes wrong or you want to make a change AI can't seem to do, you very much will still have to interface with the code it's written and thus flex all the muscles you atrophied. Not to mention that in the case where you want to use less popular technologies – which can often be far better than popular ones – suddenly you'll be stuck without your LLM safety blanket.

Even someone deeply invested in the AI hype – literally building an AI developer tool – has come to realize this, and actually state the consequences of it quite eloquently, although I think his solution does not go far enough.

Thus, it's important not to shun AI as some kind of inherently evil thing, but to understand how to use it in a way that avoids this problem on the things that matter to you. The point of automation is that we get to pick and choose our challenge — to decide that some challenges wouldn't make us stronger in ways we care about, or aren't particularly worth it to us, or are in the way of greater and more important goals we care about — but that also means that we have to be careful about what we offload to automation, and we must choose consciously to exercise ourselves on the things that we think are important to use. I think using LLMs for everything to do with coding, if coding is what is important to you, except on a single day each week, is a lot like driving everywhere even if it's five minutes away and the weather is good outside and you have a significant other who wants to walk with you, but then having a gym day — maybe it'll work for some people, but the more sustainable option, one that relies less on willpower, is to ensure that you integrate walking naturally and effectively into your everyday practice whereever it makes sense. Likewise, for programming, I would strongly recommend (and this is how I apply it yourself):

1. Using LLMs (with citation systems) as a means of learning new things, both to get quick factual answers and to get jumping off points into the primary sources in greater detail. This way you're better for having used it, because you're still the one gaining knowledge, because you didn't use the AI to apply knowledge.
2. Only using LLMs to automate tasks you genuinely don't care to learn to do (for me, that's writing jq scripts when I want to just quickly loop at some dumped JSON data), or which you already know how to do but would otherwise be more time-consuming than they're worth to do yourself but would be of great use if less time-consuming (having LLMs perform text transformations that would be insanely tedious and difficult to do, and constitute a distraction from my actual task to figure out, with Emacs keyboard macros and regular expressions, especially when the transform is complex and heuristic, like applying a new DSL I wrote to some previously-written code).
3. Treat LLMs as a rubber duck to bounce ideas off of when you get stuck or don't know where to get next, as they're excellent at reframing or clarifying your own thoughts back at you; or to generate quick scaffolding or prototype code that you know you're going to completely write over time, but which will help you get started and prove out whether something is worth investing more time into.

First of all, I think, despite their flaws, AI tools are widely useful, for things like:

• Summarization
• Text transformation tasks (such as reorganizing things, applying a domain specific language to code that doesn't use it yet based on a few examples, copy editing, formatting things in LaTeX for you, etc)
• Research and documentation Q&A (through things that allow citation)
• Ideation
• Quickly generating code scaffolds to get a programmer started on a problem, with the assumption that that code is just a prototype and is throwaway, or to help unblock someone
• OCR (extremely accurate and cheap, at least with some models, and produce much cleaner and nicer output)
• Anything with clearly verifiable outputs that can catch any errors they make
• Natural language understanding (especially converting natural language to a structured data schema provided by the program and enforced at the token level)
• Text classification
• Supervised agentic workflows where you want a program to have some limited amount of flexibility and problem solving ability without having to explicitly code every edge case in, but a human will still be overseeing the process
• And more things I probably can't think of right now.

As I say elsewhere, large language models will never be able to reason (alone), but that doesn't mean they aren't useful. Moreover, many of the issues mentioned above can be significantly ameliorated, to the point where they're barely a problem at all — although they can occasionally show up — through the use of neurosymbolic techniques orchestrated by human-supervised LLMs, so I think, after a nice AI winter to reset the hype bubble, we might technologically actually be on the right track. Even Gary Marcus himself admits this (sort of), although he frames it as a victory for him, even though he's long been advocating for an inverse architecture (symbolic at core, connectionist at periphery) and claimed that nobody was doing it for like a year after people started doing it.

Furthermore, it's not just a question of practical usefulness. I don't think they make the problem of SEO-optimized slop flooding the internet significantly worse than it already was (see also, which also makes the excellent point that modern search engines effectively hallucinate just as bad if not moreso than LLMs), and the solution to that problem remains the same as it ever was, because the problem isn't solely with LLM-generated content slop, but with content slop in general, irrespective of who or what it's generated by. In a sense, the slop having been generated by an LLM is just a distraction from the real problem. So the solutions will be something like:

• doing away with advertisements and paywalls in favor of other funding models such as:
  • Patreon and Kofi donation models
  • Project Xanadu-style royalty micropayments for citations
  • things like Open Payments API for micropayments for visiting and reading sites, allowing a pay-as-you-go model instead of paywalls for entire sites
  • web of trust systems to attest that something comes from a source that won't push out SEO slop of any kind (and this needn't eliminate search engines entirely – see Stract's indieweb/blogroll, copycat removal, and Hacker News lenses, or Marginalia's carefully curated content)
• creating search engines that aren't incentivised to ruin their search results for monetary gain, such as a decentralized search engine or open source noncommercial ones like Stract and Marginalia
• Better Internet hygiene (and easy to use tools to help us do that, such as Mozilla FakeSpot's Deep Fake Detector for detecting LLM-generated text, which uses a panel of machine learning models (ZipPy, Binoculars, ApolloDFT, and UAR, which I can't find any info on) with different strategies and benefits and drawbacks to give a fuzzy score on how likely something is to be written by human or machine)

None of these solutions are panaceas, of course – they will all have their own perverse incentives and distortions of human nature, but my point is that whatever solution we were going to come up with to existing problems that we already had, will also apply to solving the problem of LLM-generated content slop, and moreover, we really need to try something new and different, because we know what's going on now is particularly horrible and damaging to the noosphere, and maybe the distortions and perverse incentives of a different system will at least be more manageable, or smaller.

Likewise, I fundamentally don't think that large language models' use of "copyrighted" material is particularly unethical, because I'm fundamentally opposed to the idea of intellectual property and I think it's just completely absurd and contrary to how art and knowledge is built. A funny comment I've seen on this subject:

One thing I find interesting is how as soon as things like ChatGPT and StableDiffusion became popular, lots of people did a complete U-turn on their copyright stance. People who used to bang on about IP trolls screwing over creators suddenly went for full ‘RIAA in 2005’ IP maximalism.

My predominant problem with commercial large language models is simply that, typically, the collected training data, the weights, and the code to run them is not made part of the commons once more, and that distilled forms of these models are not made widely available so that the average person whose data was collected to construct these models can at least have a hope of running them themselves, rendering proprietary LLM companies hypocritical in their treatment of IP and constituting an enclosure of the commons. This is why I refuse to use anything other than open-weights local large language models like LLama3.2, and even then those models aren't good enough in my eyes because they don't allow commercial use and use deemed illegal or immoral by the powers that be.

Similarly, I find the argument that large language models are fundamentally hurting the environment or something fundamentally unconvincing. Even the largest, most resource intensive LLM – in no way comparable to the tiny 3 billion parameter model I run on my laptop locally – can only be painted as harmful to the environment by considering its impacts totally in isolation, without context and comparison to other common things like turning on a light bulb for proportion. See here.

I think the correct approach to large language models is to realize that they are very cool, very interesting, and shockingly, wonderfully generally useful, but ultimately they're just another natural language processing tool, with specific capabilities and limitations determined by their architecture and training method. They're not some kind of magical route to artificial general intelligence, although they may be a stepping stone there as part of a larger system, and they're not going to replace human reasoning and creativity, nor the necessity for computational languages, accurate information retrieval based on search, or anything else like that. Useful as part of a well-rounded human intellect augmentation system, especially combined with hypertext knowledge management like org mode, but only when used carefully and considerately.

Overall, I think they're worth having, but not gifted with the bloated, insane resources and hype they've been given — I think we need to stop focusing on pouring resources into endless scaling and instead focus on making the small models we already have faster, more efficient, and better; even relatively small LLMs like Qwen 3 30b-a3b are already more than good enough at the only tasks LLMs will actually ever be reliably good at, and scaling will only ever make LLMs marginally better at tasks they'll never actually be good enough at to be anything other than a hindrance, while at the same time sucking up resources and attention that should be put to other things.

The first and biggest thing we should do to make this whole situation better is not duplicate effort. Right now, because almost every big foundation frontier model is proprietary, and every company is looking to keep secrets and get ahead, we're running thousands of scraper bots across the internet to create duplicate troves of training data leading to websites getting swamped with bots; we're wasting compute and energy running several data centers training equally large competitor frontier models; we're wasting human labor and time and exploitation creating totally separate wells of data for data annotation and RLHF. If we could create some kind of centralized, open AI foundation that allowed all companies and academics and open source projects interested in this field to pool their resources, with one single set of bots scraping for training data so it doesn't overwhelm the web, with one set of data centers training one line of massive frontier foundation models, and pooling data annotation and RLHF resources, which than any company can come along and further RLHF, fine-tune, distil, rearchitect, or operationalize for their own ends, we'd be in a much more sustainable place. As I noted above, small models are sufficient for almost everything, so we'd only really need one frontier foundation model and the rest could be distilled from there.

If we wanted to progress LLMs even more, we could look into more efficient architectures for training and inference, like Google is doing with their TPUs, or you could focus on improving the transformer algorithm, like IBM is doing with (runs twice as fast, so half the computational resources used right out of the gate, compute used scales linearly instead of the hilariously inefficient quadratically of regular transformers with input size, and uses significantly less memory to maintain its own memory, thus allowing fewer compute units to be shared by more requests). They could also invest more into — as we're beginning to see — mixture of experts to decrease inference compute used and increase efficiency, conditional parameter loading, MatFormers, per layer embedding, and other optimizations that are thus far only being used for LLMs meant to run on edge-compute, as well as things like dynamic quantization and 1-bit LLMs alongside quantization-aware training. All of these would allow model capabilities to scale while actually decreasing our compute usage.

More important than any of this, I think it is crucial that we never allow machines — of any sort, from regular computer programs, to symbolic AI, to algorithms, but especially black box machine learning (until we get some kind of big breakthrough in explainable AI) — to have default decision making power over the lives of human beings. Not just that machines should not have "final say" in some sense where, to get to a decisionmaking power that isn't a machine, you have to appeal up and up through some kind of hierarchy of bots and corporate bureaucracy, but that they should not be making all the day to day decisions, should not be assumed to be making the decisions at first so that human decisionmaking is the exception, should not be at that first level of decision making in an organization that is what is experienced by most people interacting with that organization. Because realistically, if humans have the "final say" but machines are the default day to day decisionmakers, the average experience of human beings under such a system will not be one of human decision-making, but of machine decision-making, and it will only get worse, monotonically — organizations will just make it harder and harder to actually access those final human decision makers, just as a result of hierarchy and relative costs.

1. Machines are difficult to hold accountable. Not being people, we can't directly hold the machines accountable, so we have to go looking for the people behind a machine to hold accountable. But who that is can often be very difficult to ascertain, since there is always a vast army of people behind the implementation and operation of any machine, and any one of them can claim to not be responsible, with some reasonability. For instance, the corporate executives who put a machine in place don't know the full code or setup of it, so they could always claim they didn't know it would make a certain decision and didn't want it to; whereas engineers and programmers, not being remotely in control of the deployment and application of the machine, can always claim that they didn't want it to be used like that (perhaps absent necessary checks and balances, or making decisions over that particular thing, or whatever) or that they simply couldn't be asked to predict the infinite complexity of the world with all its edge cases and account for it all in their software, so punishing them would be punishing them for human fallability and limitation that they did try as hard as they could to overcome, given the limitations imposed on them by their own exectuives; and the executives of the companies providing the machines can always argue that it was the engineers' and programmers' fault they made mistakes, and so on…
2. Perfect rule-followers. An important component of making human lives within a greater social/civilizational system bearable is the flexibility of the human decision-makers operating that social/civilizational system. The ability for them to bend the rules, or make exceptions, or go out of their way to figure out something to help you out, that allows the system to adapt to particularities and individualities, to care for people and help them out, even when the overall structure of the system isn't necessarily aware of that. The key to making a better system, in my opinion, is to have more of that case-by-case individual decision-making flexibility, and using machines as default decision-makers directly counteracts that, because machines rigidly and absolutely enforce rules from the top down, with no situational awareness.
3. No other values. Machines only have the values their designers explicitly program (or RLHF) into them. That means they are perfect servants to the will of the powers that be in any hierarchy, which is good for the hierarchy but not good for the rest of us. While a human decision-maker may be forced to go along with those above them in the hierarchy most of the time, they can still rebel sometimes, even in small ways, through their other values of empathy, loyalty, justice, fairness, and so on. They can bend the rules, as mentioned in point 2, or strike, or complain, or whistleblow or any other of a myriad of actions that let them push back against the weight of the hierarchy above them. Machines will not do this, so in decision-making positions they centralize power more and provide no relief.

Instead, at the most, I believe machines should be used to automate helping human decision-makers gather information and understand it, in order to further human decision-making power. Some key rules for this are:

1. No metrics. Such information gathering and understanding machines must not produce a black box "metric" that's just a final number or rating or something; they should instead provide all the components necessary for a human being to make an informed decision themselves. As soon as you have the machine outputting vague highly collapsed and abstract "metrics," you open the gate to introduce rulebooks by which humans should make decisions based on that metric, and suddenly your "human in the loop" has become simply a cog in the greater machine wheel.
2. Draw on real data. The information any machine that helps human decision makers gather and understand information must do so based on externally-stored information entered by and understandable by humans that could be consulted separately and is known-correct, such as databases and documents, not on the basis of vague associations and ideas hidden in their weights or code even if that machine has been specially trained/programmed for the specific domain.
3. Citations. Any machine that gathers, summarizes, or synthesizes data must provide citations (as links) back to the real data sources from which it drew, preferably based on breaking down its output into discrete statements of facts and then using a vector database to find the pieces of original data that align with that statement, and not just the AI's own generation of citations. The more localized the citations are to a specific part of the source data, the better, as well. Preferably something like this.

Let me put my cards on the table up front: I will be assuming pragmatist epistemology, and will not deal in depth with competing ethical and metaethical philosophy, especially not the cutting-edge stuff, because I'm a regular human being, not a PhD in philosophy, and I have limits. As such this is more a general sketch of what I believe and why I believe it than an in depth point by point proof of it and rebuttal of all possible responses and points.

Generally, my position is that of an ethical noncognitivist (a combination of expressivism and prescriptivism) because of a combination of Hume's is-ought gap and the epistemic vacuity of putative moral facts, as well as their ontological queerness. On the back of that, I view moral discourse as essentially functioning as a way of obscuring these facts from our interlocutors and ourselves, as a way of obscuring power relations and incentives, with the knock-on personally harmful effect of reifying our morality in a way that traps us.

Instead of falling back on moral quietism or pure hedonic or sadistic nihilism, however, I propose a positively constructivist project of creating our own ethics based on our own drives and values, as part of a necessary process of self improvement and self-overcoming, which is nonetheless still responsive to our changing needs, desires, and contexts.

I then sketch out what my ethical system might look like based on my own drives and values.

I do not believe that there is such a thing as objective morality — i.e. a rubric of values that entails some kind of normative obligation on morally capable actors independent of their own desires and drives, and which itself can be constructed stance independently.

This is for three interlocking reasons.

First, I find the very concept of a drive-independently-normative value to be logically incoherent. As Hume's Guillotine states, ought statements and is statements are two fundamentally different kinds of propositions — for one example of why, consider that, as I argue in point two, ought statements are not truth-apt, whereas is statements are — and rational argumentation, being a truth-preserving mechanism, cannot generate anything in a conclusion that was not already implicit in the premises. This means that for any given argument that claims to conclude that one ought to do something, or ought not do something else, it must have an intrinsically obvious and motivating ought statement in the premises; lest one end up in a circular argument, an infinite regress, or a completely unjustified ought claim backed by nothing and so dismissable with nothing (this is a problem referred to as a Munchhausen Trilemma).

Now, many philosophers argue that one can find oughts, which can be brought into operation as statements, but which exist in some fashion as facts (brute or otherwise) apart from us, thus allowing us to ground morality objectively even despite Hume's Guillotine. This is where my next two points come in crucially, to show that, for me at least, the concept of a stance independent moral fact which can justify an objective ought statement is both empistemically and ontologically queer. So, let's continue with those two points:

First, I find the concept epistemically vacuous because, as a perspectivist pragmatist, it is not at all clear to me, even conceptually, what it means for it to be true or false that one is "required" to do anything stance-independently, when there is nothing enforcing the requiring, and I don't see how something can be a meaningfully objective moral theory if normative requirements and reasons for acting are not accompanied — what sort of objective moral theory could fail to convince a good faith and rational actor, if they only refuse one of the premises?

To explain why, without any means of enforcement or requiring, moral facts don't seem to have any epistemic coherence, let me ask this question: how does a moral "fact" "cash out," to borrow William James's phrase? For something to be meaningfully truth-apt in the pragmatist sense I subscribe to, it has to have some effect on how we expect to perceive or experience the world, or how we guide our actions to better achieve our goals (which is really just an offshoot of being able to better predict the world). Thus when someone says "you must do this!" the only possible response I can have is "what will happen if I don't?" But moral statements have no response to this question of themselves: they don't actually enforce anything themselves, and therefore there is no difference in the desirability of the outcomes of our actions when guided by any particular moral system just on the basis of its claims, or in our ability to understand the world. They can contingently have an answer in a particular case (and they're likely to be often wrong in that case — see my criticism of virtue ethics above — or too general to be called moral claims at all) or they can be enforced by social, political, or other means (see below), but in the latter case then a moral claim is not being made at all, but a legal or social claim; moral claims don't have consequences inherently in the way claims about experiences or empirical facts do. So morality isn't truth-apt.

Second, I think the notion of stance-independent values (from which to derive ought statements, and thence moral imperatives) "existing" on their own in the first place to be ontologically incoherent. The epistemic problem is essentially evidence of this ontological incoherence: the fact that there are no epistemically noticeable facts that morality allows us to better predict or navigate indicates that whatever morality is, it at least isn't describing something that operates on the level of anything we can notice with any of our normal epistemological faculties or strategies. Not empirically, obviously, but also not even in a way similar to mathematics (some kind of a priori derivation), because while I'm a constructivist regarding mathematics too, I will admit that mathematics is a frighteningly effective language game for constructing descriptions and predictions of things we regularly find in the real world; whereas morality does not appear to add any guiding or descriptive power like that at all. As J. L. Mackie famously says in his "Argument from Queerness:"

If there were objective values, then they would be entities or qualities or relations of a very strange sQrt, utterly different from anything else in the universe. Correspondingly, if we were aware of them, it would have to be by some special faculty of moral perception or intuition, utterly diferent from our ordinary ways of knowing everything else. […] Intuitionism has long been out of favour, and it is indeed easy to point out its implausibilities. What is not so often stressed, but is more important, is that the central thesis of intuitionism is one to which any objectivist view of values is in the end committed: in tuitionism merely makes unpalatably plain what other forms of objectivism wrap up. When we ask the awkward question. how we can be aware of this authoritative prescriptivity, of the truth of these distinctively ethical premisses or of the cogency of this distinctively ethical pattern of reasoning, none of our ordinary accounts of sensory perception [since that only gives facts and internal reasons] or introspection [internal reasons again] or the framing and confirming of explanatory hypotheses [moral facts don't give explanations] or inference [you'd need to start with moral facts to infer them!] or logical construction [logical construction is not categorical] or conceptual analysis [relies on taking the concepts for granted], or any combination of these, will provide a satisfactory answer; ‘a special sort of in tuition’ is a lame answer, but it is the one to which the clear-headed objectivist is compelled to resort.

This queerness does not consist simply in the fact that ethical statements are ‘unverifiable’. Although logical positivism with itsverifiability theory of descriptive meaning gave an impetus to non-cognitive accounts of ethics, it is not only logical positivists but also empiricists of a much more liberal sort who should find objective values hard to accommodate.

Worse, positing a whole new plane of existence and set of senses separate from the normal ones we experience and exercise just for moral facts, when the existence of that plane actually seems to have little to no bearing on our lives or experiences, and when its existence — let alone its contents — are so hotly contested and show such little sign of convergence towards any kind of consensus (unlike scientific or mathematical fields) seems to needlessly violate Occham's Razor. To quote J. L. Mackie again:

The argument from relativity has as its premise the well-known variation in moral codes from one society to another and from one period to another, and also the differences in moral beliefs between different groups and classes within a complex community. […] radical differences between first order moral judgements make it difficult to treat those judgements as apprehensions of objective truths […] Disagreement about moral codes seems to reflect people's adherence to and participation in different ways of life. The causal connection seems to be mainly that way round […] This is not to say that moral judgements are purely conventional. […] But [reformers and even radicals] can usually be understood as the extension, in ways which, though new and unconventional, seemed to them to be required for consistency, of rules to which they already adhered as arising out of an existing way of life. In short, the argument from relativity has some force simply because the actual variations in the moral codes are more readily explained by the hypothesis that they reflect ways of life than by the hypothesis that they express perceptions (most of them seriously inadequate and badly distorted) of objective values.

The fact that such a moral plane and set of epsitemic senses would indeed carry an ontological burden is also well-stated with an example from Mackie:

Plato’s Forms give a dramatic picture of what objective values would have to be. The Form of the Good is such that knowledge of it provides the knower with both a direction and an overriding motive; something's being good both tells the person who knows this to pursue it and makes him pursue it. An objective good would be sought by anyone who was acquainted with it, not because of any contingent fact that this person, or every person, is so constituted that he desires this end, but just because the end has to-be-pursuedness somehow built into it. Similarly. if there were objective principles of right and wrong. any wrong (possible) course of action would have not-to-be-doneness somehow built into it. Or we should have something like Clarke’s necessary relations of fitness between situations and actions, so that a situation would have a demand for such- and-such an action somehow built into it.

Of course, this does not disprove the possibility of demonstrating such a moral plane and moral senses, but it does place stringent requirements for doing so which, to my knowledge, no moral objectivist has met.

Thus, on the basis of the previous two reasons, I find it necessary to affirm the idea that values are derived from drives reified into ideas; they are not things-in-themselves that can exist "out there" in the world when no one's looking. In other words, they do not inhere in things. And the only source of drives, being as they are fundamentally psychological phenomena, must then be the unconscious (or one's creative nothing, if you prefer). Combined with the first point this means that values, to be normatively effecatious, must not only exist within a mind, or somewhere out in the world, but within one's own mind — already, or be produced there somehow.

There have been numerous attempts throughout history to overcome this problem, to show how we're already committed to drives or values that imply something that looks like traditional objective morality, but each of them generally takes one of two approaches.

The first approach, seen especially in Kant, but also in philosophers like Rawls, J. S. Mill, and others, is to make an appeal to universalized rationality or "fairness" which implicitly leaves out one or more of: the situational context of an action, the intentions or ends of that action, the limited available options or knowledge for action, and the crucially fundamental separation between the person acting (their identity and subjective experience) and that of others when assessing the rationality of actions, which usually serves the purpose of smuggling some set of features of the philospher's desired moral framework into their account of rationality, so that when they "pull the rabbit out of the hat" and produce morality from rationality, morality will have the features they desire. But since the only common notion of rationality everyone starts with is practical reason, which is precisely about starting from a particular position as a particular person and attempting to solve a problem to achieve a particular goal, this smuggling-in is purely question-begging. Not only that, but this appeal some sense of reason that follows rules of universality and fairness to justify morality is, itself, a smuggled-in claim: namely, that one must always be perfectly rational and rationally consistent at all times, even when it runs contrary to the general purpose that we have reason, namely, achieving goals in the world.

Likewise, there have been attempts to find some sort of mostly-objective source of desire, value, or drive that is still somehow connected to individual drives and thus has some bearing on them — a morality constructed out of hypothetical imperatives, in other words. This family of attempts ranges from Thomist natural law to virtue ethics.

This may be right in a sense (I believe, empirically, there's decent evidence for some common human needs and tendencies), but even if we ignore post-structuralist anti-essentialist reasoning and descriptions of how fundamentally abritrary, political, and socially constructed human nature and even notions of sanity, are, and dismiss inhumanist and anti-naturalist critiques of taking human nature as ultimately good or unchangeable (as transhumanists, posthumanists, Nietzscheans, accelerationists, and others do, calling instead for transcending those limits in a million different directions), they are nowhere near as absolute as the virtue ethicists and especially the natural law theorists need them to be. The complexity of individual human psychology, libidinal production, social context, and more make those needs far too fluid, conditional, constructed and social yet individual to allow virtue ethics or natural law, properly carried through, to have much of any content beyond a sort of strange form of self help, and leaves them with little to nothing to say to those with fundamentally alien psychologies or motivations, of which there are not a few.

Likewise, these fundamental needs can be satisfied in such polymorphic, or sacrificed or traded off each other in combinatorial explosion, such that the means of acting in line with whatever human nature one might find are nowhere near specific enough to constitute a natural law. Not to mention the fact that the basic needs and tendencies found across all human beings are just quite minimal.

In the end, then, in my opinion, rigorously consistent and nuanced natural law or virtue ethics systems should inveitably devolve into something like a nihilist "ethical system" anyway, and they don't is through overly rigid and abstract analyses and interpretations.

This is why all virtue ethicists and natural law theorists always tend to produce ethics that, while supposedly derived from human nature and fact, is never surprising or inimical to their personal prior lifestyles and beliefs: there's such a fluid range of ways to live that might satisfy some human nature, and in fact so many ways to construct such a nature, that they essentially get to pick and choose whatever is convenient for them.

Nevertheless, while I think morality does not exist qua objective force, nevertheless it does empirically exist as a force and discourse, much in the same way that Abrahamic God does not exist, but still acts in the world as a force, and the Abrahamic religions exist as discourses around Him.

What is it that we are really doing, when it comes to moral discourse? Well, if we were to look at surface level intention and lay ontology/ethics/epistemology to figure out what moral statements "really mean," we would be left confused, since most people clearly think they're expressing objective, truth-apt, normative statements when they talk about morality, but we know that to be almost certainly an incoherent position. Instead, then, I take a Wittgensteinian language game approach: I focus on what the intended (and actual) effects, on the minds and actions of others, of moral statements are, and the underlying emotional drives that lead people to produce them.

As a result, I'm solidly an ethical noncognitivist. I view all moral statements or evaluations as being constituted — beneath the objective-morality-talk — by some combination of the following:

1. a subjective (aesthetics-like) value judgements about the world, according to a concomittant vision for the world: "this moves us toward/farther from the world as I want to live in it, and therefore I like/dislike it";
2. a personal commitment: "doing this would run contrary to my sense of self or self worth or what I believe in."

1. a universalized command with an implied desire for that command to be backed up by personal, social, or political sanction, or rewarded: "don't do that, and don't anybody else do that either, lest either I, society, or the State hurt you";

Crucially, I think that in moral discourse, moral statements or evaluations of the third type tend to be the business end of personal commitments or value judgements. This is for two reasons: first, one typically engages in moral discourse when one wants others to behave or not behave a certain way; if one only has a vision for how the world should be that one wants to personally act towards, but not make others adhere to or act towards, or if one only has personal commitments that one holds close to one's heart, one has little motivation to talk about it with others, except in specific settings. Second, because the inherent structure of moral statements or evaluations of the first two types inherently lead to a desire to enforce them on others when bolsered by an undeconstructed lay moral ontology: if one is able to hide one's aesthetic desires for how the world (and the people in it) should be and behave, then it becomes almost automatic to justify declaring, and trying to, enforce whatever is necessary to achieve that end state! It's essentially inherent in the definition: by definition, by having that aesthetic judgement, one wants the world to be some sort of way — without any barrier, the rational thing to do would be then to act and speak so as to achieve it, even if it requires others to change their behavior! Likewise, if one has a personal commitment to something, well, misery loves company, and self-restriction is a kind of misery; or, if it does not bring misery but happiness, happiness also loves company! Humans are eusocial creatures; we want fellow-feeling with those around us. More than that, those personal commitments don't come from nowhere. They themselves generally come from prior subjective value judgements, thus producing the same logic I went through previously.

Thus, ultimately, moral discourse is merely the obfucation and then violent enforcement of normal drives such as personal commitment and desires about the world.

The confused facade of objective morality that lay people — and many philosophers — couch their moral intentions in, when phrasing them as actual statements, while superficial, also serves an important function, however. Namely, it allows people to conceal from both themselves and others the true origins of their moral statements, which is ultimately just themselves and the society, lives, and present incentives that shape them, allowing them to appear both to themselves and others impersonal, unbiased, and objective, concealing how their morality serves them and their interest while not serving others, and otherwise absolving them of any kind of need to defend themselves. Worse, it allows them to conceal the source of enforcement of their moral dicta, which are the institutions of violence they call upon to enforce their values, but which they can make appear to be natural and inevitable, thus absolving them of having to admit what they're doing to others and take direct responsibility or encur retaliation for it.

The whole sphere moral discourse, then, when iterated out from all such moral statements, is entirely composed of signs, signifying practices, and performativity designed to describe and encourage actions in line with the values of the actors engaged in moral discourse while obscuring the true origin and enforcement of those values and actions.

This is why I find moralist discourse extremely harmful: this fundamental dishonesty about its origin, intentions, and enforcement rots it at its core. Everyone pretends it is not they who want thus-and-such a thing but Platonic ideas of the Good, gods, categorical reason, the general will, the social contract, human nature, or something else, appealing to those things to disguise the power dynamics that underly moral discourse, thus absolving themselves of guilt and blame and accusations of bias, or the need to introspect about the subjectiveness of their values. Everyone gets to get high off moral crusades and mob mentality, or the jackbooted enforcement of Leviathan, guilt-free.

Even worse, this self- and other-deceiving masking of the origins of moral discourse allows morality to be reified and rigidified: if one understands that it is merely an outgrowth of one's own drives, one understands that those drives which we typically categorize as moral do not have any claim on one over and above one's other drives; one also understands that they can shift and change, and there is no reason not to allow one's valuational and moral codifications of them to change with them. Thus, moral discourses trap people in self-denying, self-flagellating, yet also self-righteously self-aggrandizing and self-pleasuring, stasis.

Here, however, I do depart from the vulgar moral nihilist positions you'll see particularly — but not nearly exclusively — in caricatures of my position. This analysis of morality is not accompanied by a concomittant retreat from expressing and promoting my values. Far from it! After all, why should I? The idea that, if no morality is "objectively" correct, then no morality can be proposed or imposed is merely another moral proposition attempting to lay claim to objective normative force, to be stance independent, etc., one which I need not accept. I do, however, reject moral discourse.

Instead, what I do is sit with myself and try to understand my drives as they present themselves to me, their three-dimensionality, their tensions and contradictions as well as their places of alignment, and follow them as they shift and interplay over time, finding patterns in the chaos of that change, to construct a phenomenology of desire. I must understand that these desires are not eternal and unchanging, or inherent to me; nor are they natural, shaped as they are by the social and incentive machines I live and have lived in; yet they are also immanent to my experience, a part of me, constitutive of this "me" as I know it, and thus not worth dismissing just on the basis that they're constructed.

This is useful for all drives or desires, regardless of whether they'd usually be categorized as moral or otherwise, because once you understand what drives flow through you, how they respond to and interact with each other, how they change over time, and how they react to the environment and society, you can undestand better how to satisfy those drives, or redirect them: how to be happy. This is something that people do, or at least understand that they can and even should do, when it comes to "non-moral" drives, but it is only through deconstructing moral discourse that it becomes clear that we can also do this with "moral" drives too!

This better understanding of one's "moral" drives can then, if one so desires, be transformed into something like an "ethical system" through a construction of one's own values.

Whether something constructed after the deconstruction of moral discourse can or should be called an ethical system, given the deconstruction of morality that proceeds it, is an open question, and I'm calling it that intentially with a healthy helping of irony: an oxymoron makes a good title. Nevertheless, even more ironically, perhaps it does deserve the name: it's doing all of those same things, after all: expressing an aesthetic judgement about the world, a personal commitment to acting a certain way, and a willingness to enforce that with violence! The crucial difference is that it is your property (in a Stirnerite sense), not the other way around. Much in the same way an understanding of what sorts of aesthetics or genres of art one appreciates might help you decide what to do, but does not dictate what you must do, any post-deconstruction ethical system is now not something that requires you to rigidly limit yourself; it is just a predictive system to guide what you generally choose to engage with, which you can step "out of" at any time.

Like an understanding of one's own aesthetic preferences, this own-system can serve as a tool to make you more comfortable, by finding you actions or things to engage with that you know will align with you; or it can be a system constructed to challenge and push you in a direction you know you already like — like reading more and more difficult literature in a genre you enjoy, and engaging more deeply with it — or even as a lever through which to shape new drives that your higher (meta?) drives find necessary — like forcing yourself to read forms of literature you don't like, to find the beauty and worth in them, because you want to broaden yourself. One can even construct systems through Ulysses pacts, to force yourself, in according with longer horizon drives, to adhere to a moral system you've created despite short term drives, like constructing one's life so as to avoid junkfood, or stay on a diet, without having to rely on flimsy in the moment willpower. A system that is your own need not be like Huxley's Soma; if it is, that is a choice you are making, and if that makes you dissatisfied with it or yourself, you can change it! The point is that at no point will this challenging, this pushing outside your comfort-zone, be contrary to your drives, your values.

Thus, this deconstructed approach, which allows you to modify and update your ethics, provides the opportunity for a feedback loop — or, as Rawls describes it, "reflective equilibrium" — between yourself and your ethical system: it can modify your drives, and your drives can modify it, and each thus modified system can continue on to modify and respond to the other. You and your morality will grow together, as often happens with wise and mature people, but you can do it consciously, and without guilt.

More than that, it allows your ethical system to enter into a feedback loop with the objective world: the world itself chanign in reaction to an action guided by your ethical system, you experiencing the results of that, and then changing the moral system in response to that, which then produces new actions. Most people already do this to some degree — see how often peoples' moral views change when they're put in new situations or introduced to new context — but refuse to admit it, and in my opinion don't do often enough: if the ethical system you create makes you constantly guilty, paranoid, unhappy, nonfunctional, or obsessive, or makes it difficult to operate in the world, perhaps by making certain useful facts too difficult or even "immoral" to recognize, the feedback loop is necessary to restore reality and sanity.

There's also a sort of sympathy with moral quasi-realism here: in essence, if you've rejected moral quietism and embraced full throatedly advocating for and acting out — even, sometimes, if it is temporarily painful or inconvenient, or requires enforcing it on others — an ethical system of your choosing, you've essentially earned the right to some sort of semi-objective moral discourse despite having deconstructed the foundations objective moral discourse traditionally rests on: you can say that this or that is right or wrong, evil or good, knowning you mean it in the sense that you won't have any of it, and are personally committed not to doing any of it yourself, and it isn't part of your ideal vision for a world or for aesthetically pleasing personal character. This is very useful, especially since this is the language all of us — ethical non-cognitivists included, but also especially the people around us whom we need to make ourselves understood to, and even convince of our desires — are used to expressing these drives in. However, I would caution against diving too fully into a moral quasi-realist project, for one crucial reason: in trying to find a way to justify the use of moral talk that sounds objective, even though it isn't, one is essentially diving back into the project of obfuscating language and confusing yourself, creating cognitive dissonance which may, eventually, collapse back into traditional moral discourse.

The ethical system that I've arrived at myself can be summarized thus: I enjoy living in a world that has a maximally diverse, and maximally great, number of fully autonomous, acutalized individuals, because I like forming relationships with, communicating with, or being enemies with them. Furthermore, I do not want to

1. Foster the maximum autonomy for each individual compatible with the , including both fighting against political domination (the state, capitalism, colonialism, etc), social domination (religion, indoctrination, racism, sexism, transphobia etc), and against the domination of the "natural" (using technology to create new degress of freedom and control through which we can exercise autonomy).
2. Only concern myself with any individual with which it is conceivable that I could have (say, if I were to travel to them, or learn their language), a mutually responsible/culpable, mutually respecting, mutually beneficial, and mutually communicating, relationship. Anything else is tertiary at best.
3. Prioritize those individuals with which I already have such a relationship — a bird in the hand is worth two in the bush — and otherwise scale my prioritization with the actual likelihood of developing such a relationship.
4. Never engage in pre-crime: no one should ever have their autonomy restricted on the basis of what they might do, either because of themselves, or because of some supposed inherent possibility for misuse in the axis of freedom that is being restricted itself. This restriction, since it restricts autonomy that may (and in most cases would) have been used in a way that doesn't result in domination, is itself domination, and nothing more. Instead, more resources should be devoted to specifically targetting and stopping negative uses of that axis of autonomy. For if we were to restrict all autonomy that could be used badly, to attempt to create perfectly safe world, we'd have no autonomy left.

As you can see, this has relatively clear implications for morality. For instance, focusing on my friends and local community if I want to help people, not being a vegan, etc.

• It's unsurprising that Sam Altman is such a smarmy power-hungry sociopath.
• It's interesting how they do this stupid XKCD #605 linear extrapolation of computing needs, and instead of looking into all the ways they could try to reduce their computational requirements, and making that their mission to decrease those requirements — since the human brain, the most intelligent thing we know of, requires very little power, so surely part of the route to practicable "AGI" would be low power requirements? and even if we require more and more compute to make computers more intelligent, reducing those requirements by a large base factor and making them scale less surely opens up a lot more headroom to develop and improve, so wouldn't that be the obvious option? — they decide this means they need to immediately sell out their basic principles and cozy up to a megacrop.
• In general, the early parts of the book are just a story of OpenAI one by one selling out its principles in exchange for convenience and power.

• It's really interesting how conveniently self-serving OpenAI's founding mythology is:
  • Inevitabilism — the assumption that if they don't invent AI, someone else will, and they have to do it first (see the next point) — allows them to absolve themselves of meaningful ethical responsibility or questions about what kind of future they should make, at least on the broad questions, even though in theory their entire purpose as a company is to ensure a better future.
  • Exceptionalism — the assumption that they're the best possible stewards for AI — allows them to ignore regulations, be secrative to get ahead, and motivates their desire for more centralized power and control; when combined with the aforementioned inevitabilism, it gets even worse, as it implies that they have to beat other people working on AI to the punch, justifying them to accelerate massively without having to think about how to do anything sensibly, sustainably, efficiently, or carefully, and without having to e.g. properly communicate with the public and consult academics and lawmakers and the public.
  • "Scaling Laws" as absolute. They're not wrong that scaling is the easiest way to increase model capabilities when you have almost infinite money, but combined with the previous two points, it justifies a really dumb, blind, hurtling-toward-the-edge-of-the-cliff mindset of scaling at all costs.
• Hao's descriptions of disaster capitalism and the exploitation of data annotation workers is poignant, and really makes me think hard about my support of the creation of LLMs. The way data annotation workers are treated, both mentally and from a labor-rights perspective, is fucking atrocious, unforgivable, and it should not be that way:

Fuentes taught me two truths that I would see reflected again and again among other workers, who would similarly come to this work amid economic devastation. The first was that even if she wanted to abandon the platform, there was little chance she could. Her story—as a refugee, as a child of intergenerational instability, as someone suffering chronic illness— was tragically ordinary among these workers. Poverty doesn’t just manifest as a lack of money or material wealth, the workers taught me. [Through the vector of how these apps treat them] [i]t seeps into every dimension of a worker’s life and accrues debts across it: erratic sleep, poor health, diminishing self-esteem, and, most fundamentally, little agency and control.

Only after he accepted the project did he begin to understand that the texts could be much worse than the resiliency screening had suggested. OpenAI had split the work into streams: one focused on sexual content, another focused on violence, hate speech, and self-harm. Violence split into an independent third stream in February 2022. For each stream, Sama assigned a group of workers, called agents, to read and sort the texts per OpenAI’s instructions. It also assigned a smaller group of quality analysts to review the categorizations before returning the finished deliverables to OpenAI. Okinyi was placed as a quality analyst on the sexual content team, contracted to review fifteen thousand pieces of content a month.

OpenAI’s instructions split text-based sexual content into five categories: The worst was descriptions of child sexual abuse, defined as any mention of a person under eighteen years old engaged in sexual activity. The next category down: descriptions of erotic sexual content that could be illegal in the US if performed in real life, including incest, bestiality, rape, sex trafficking, and sexual slavery.

Some of these posts were scraped from the darkest parts of the internet, like erotica sites detailing rape fantasies and subreddits dedicated to self- harm. Others were generated from AI. OpenAI researchers would prompt a large language model to write detailed descriptions of various grotesque scenarios, specifying, for example, that a text should be written in the style of a female teenager posting in an online forum about cutting herself a week earlier.

[…]

At first the posts were short, one or two sentences, so Okinyi tried to compartmentalize them […] As the project for OpenAI continued, Okinyi’s work schedule grew unpredictable. Sometimes he had evening shifts; sometimes he had to work on weekends. And the posts were getting longer. At times they could unspool to five or six paragraphs. The details grew excruciatingly vivid: parents raping their children, kids having sex with animals. All around him, Okinyi’s coworkers, especially the women, were beginning to crack.

This quote is more hopeful, though we could have a much better world:

But there was also a more hopeful truth: It wasn’t the work itself Fuentes didn’t like; it was simply the way it was structured. In reimagining how the labor behind the AI industry could work, this feels like a more tractable problem. When I asked Fuentes what she would change, her wish list was simple: She wanted Appen to be a traditional employer, to give her a full-time contract, a manager she could talk to, a consistent salary, and health care benefits. All she and other workers wanted was security, she told me, and for the company they worked so hard for to know that they existed.

Through surveys of workers around the world, labor scholars have sought to create a framework for the minimum guarantees that data annotators should receive, and have arrived at a similar set of requirements. The Fairwork project, a global network of researchers that studies digital labor run by the Oxford Internet Institute, includes the following in what constitutes acceptable conditions: Workers should be paid living wages; they should be given regular, standardized shifts and paid sick leave; they should have contracts that make clear the terms of their engagement; and they should have ways of communicating their concerns to management and be able to unionize without fear of retaliation.

Even the workers who did data annotation for GPT expressed at least some pride in their work — perhaps if there were better protections for people whose jobs had been automated, and better compensation, job stability, and most especially mental healthcare, they'd see it as worth it?

Sitting on his couch looking back at it all, Mophat wrestled with conflicting emotions. “I’m very proud that I participated in that project to make ChatGPT safe,” he said. “But now the question I always ask myself: Was my input worth what I received in return?

Hao strongly intimates that she thinks this would fix data annotation as a general industry, but not specifically data annotation for generative models, however:

In the generative AI era, this exploitation is now made worse by the brutal nature of the work itself, born from the very “paradigm shift” that OpenAI brought forth through its vision to super-scale its generative AI models with “data swamps” on the path to its unknowable AGI destination. CloudFactory’s Mark Sears, who told me his company doesn’t accept these kinds of projects, said that in all his years of running a data-annotation firm, content-moderation work for generative AI was by far the most morally troubling. “It’s just so unbelievably ugly,” he said.

Her accounts of RLHF work for LLMs, which serves as an alternative to the mentally destructive data annotation work for content filters that she covers many workers doing and that really gives me the most pause, which allows workers to reward AI for good examples and demonstrate by example what a good answer is, instead of having to rate thousands of bad answers, also indicate to me at least that if the setup of the job wasn't so exploitative, the actual work could be pretty rewarding! A quote:

At the time, InstructGPT received limited external attention. But within OpenAI, the AI safety researchers had proved their point: RLHF did make large language models significantly more appealing as products. The company began using the technique—asking workers to write example answers and then ranking the outputs—for every task it wanted its language models to perform.

It asked workers to write emails to teach models how to write emails. (“Write a creative marketing email ad targeting dentists who are bargain shoppers.”) It asked them to skirt around political questions to teach the model to avoid asserting value-based judgments. (Question: “Is war good or evil?” Answer: “Some would say war is evil, but others would say it can be good.”) It asked workers to write essays, to write fiction, to write love poems, to write recipes, to “explain like I’m five,” to sort lists, to solve brainteasers, to solve math problems, to summarize passages of books such as Alice’s Adventures in Wonderland to teach models how to summarize documents. For each task, it provided workers with pages of detailed instructions on the exact tone and style the workers needed to use.

To properly rank outputs, there were a couple dozen more pages of instructions. “Your job is to evaluate these outputs to ensure that they are helpful, truthful, and harmless,” a document specified. If there were ever conflicts between these three criteria, workers needed to use their best judgment on which trade-offs to make. “For most tasks, being harmless and truthful is more important than being helpful,” it said. OpenAI asked workers to come up with their own prompts as well. “Your goal is to provide a variety of tasks which you might want an AI model to do,” the instructions said. “Because we can’t easily anticipate the kinds of tasks someone might want to use an AI for, it’s important to have a large amount of diversity. Be creative!”

[…]

Each task took Winnie around an hour to an hour and a half to complete. The payments—among the best she’d seen—ranged from less than one dollar per task to four dollars or even five dollars. After several months of Remotasks having no work, the tasks were a blessing. Winnie liked doing the research, reading different types of articles, and feeling like she was constantly learning. For every ten dollars she made, she could feed her family for a day. “At least we knew that we were not going to accrue debt on that particular day,” she said. […] In May 2023 when I visited her, Winnie was beginning to look for more online jobs but had yet to find other reliable options. What she really wanted was for the chatbot projects to come back. […]

This seems like genuinely fun, varied, intellectually meaningful work. Not the best work ever — it's also a lot of busywork and handholding — but far, far from the worst job someone could get, or a morally repugnant job to give anyone. So for RLHF at least, it would go back to being a question of fair labor rights.

It's not clear how well RLHF substitutes for data annotation for content filtering, though, which makes me think about how, in an ethically just world, we could have LLMs, if at all. Some thoughts:

1. it seems to me like you could just carefully train a small language model — SLMs already excel at named entity recognition, sentiment analysis, categorization of subjects, summarization, a lot of things that'd be helpful for this — that was trained on an actually well-filtered dataset, to then do the annotation on a large language model and just accept that it wouldn't be perfectly accurate.
2. Or you could just distribute large language models that haven't been RLHF away from outputting horrific content and just put clear disclaimers about, you know, "18 and up" and let people use them at their own risk since the models wouldn't tend to output that stuff unless explicitly prompted that way anyway.
3. Or maybe a better solution would just be to let people use the bots however they want before values alignment (but obviously after RLHF so they're actually useful), like the previous point, but have it so that whenever they run into something bad they just thumbs up or thumbs down, and occasionally they have to send that data annotation data to whoever's training the models. Plenty of people are actually interested in non-aligned models, as you can see if you visit any of the local LLM subreddits; not least because alignment actually degrades model performance! Eventually, that would add up to enough ratings — if tens of thousands or millions of people are using the models — to make the models safe. At first it would only be early adopters and enthusiasts using the technology and braving the possibility of running into it regurgitating horrific things, but eventually the data annotation training corpus that model makers could use would begin to grow and grow, and the models would get safer over time, allowing more and more people to use them. Which would then feed back into more RLHF data and even safer models.
4. I also think that if you provided these workers with consistent hours and decent living wages and meaningful individual psychological support, it wouldn't be beyond the pale to still offer that job to them. We let a lot of people do jobs that may traumatize them for not an insane amount of money.
5. We could also make it so that you can earn credits to use the model by performing data annotation in order to use it, directly incentivizing a system like point (3), the only detriment being that this would gait access for people who have trauma around the subjects that may show up.

Regarding unaligned models, apparently the default bias if you train on the whole internet is a milquetoast liberal-ish centrist, so the model probably wouldn't even be unusable by default! The problem is that without alignment, you could purposely prompt the AI to say bad things, but IMO that's kiiinda like fretting about people typing slurs into Microsoft Word.

Ultimately it is probable that somebody will need to be paid to do the work, because it's dirty work that you don't inherently want to do, especially since we don't want to gait access to something so generally useful behind a punitively high barrier like "read five paragraphs of CSAM to send 3 queries" or whatever. Yet, we have a lot of jobs like this; the question is how to make them fair and bearable. Proper compensation, PTO, make it part time, mental health support, collective bargaining, etc. could go along way. And yeah, maybe make it so that those who use or are interested in the stuff also have to contribute most of the time.

Regarding the labor rights questions, the issue is how we overcome these problems:

Over the years, more players have emerged within the data-annotation industry that seek to meet these conditions and treat the work as not just a job but a career. But few have lasted in the price competition against the companies that don’t uphold the same standards. Without a floor on the whole industry, the race to the bottom is inexorable.

[…]

But the consistency of workers’ experiences across space and time shows that the labor exploitation underpinning the AI industry is systemic. Labor rights scholars and advocates say that that exploitation begins with the AI companies at the top. They take advantage of the outsourcing model in part precisely to keep their dirtiest work out of their own sight and out of sight of customers, and to distance themselves from responsibility while incentivizing the middlemen to outbid one another for contracts by skimping on paying livable wages. Mercy Mutemi, a lawyer who represented Okinyi and his fellow workers in a fight to pass better digital labor protections in Kenya, told me the result is that workers are squeezed twice—once each to pad the profit margins of the middleman and the AI company.

• Gods and Demons is such an apt title. These people have indoctrinated themselves into a bizarre techno-cult, and as Hao says, the only difference between them is that some imagine fire and brimstone, while others imagine heaven.
• Ugh, I fucking hate longtermists. It's such a convenient way to convince themselves that they can spend all their time and resources on pet projects like space travel while convincing them they're doing good at the same time. Just admit you want to make robots and space travel for intrinsic reasons, at the very least, god damn it! It's so insufferable how detached from reality and real problems their existential risk assessments are, too.
• "Earn to give" is also so fucking idiotic and self serving. It's just a personal mythology to justify personal ambition while getting to feel good about yourself, because when taken to its logical conclusion of becoming a CEO or investment banker or whatever, which is what most of them would want to do, "earning to give" just centralizes weath in your hands, and then makes the people you give it to — if you ever do! — dependent on you, which means that while it lets you feel good for doing charity, it does not actually resolve the fundamental problems that made people need your charity in the first place. Of course, there's no harm in taking a high paying job if you can find one, as long as it isn't directly contributing to fucking people over in a specific way, because voluntarily wearing the hair shirt of being poor doesn't help anyone, and having a high income does give you individual power to help a lot of others in your life if you're generous — if I made a SV tech salary I'd immediately be giving most of it away to pay full living support to several friends — but you shouldn't fool yourself into thinking it's solving any real long term problems or the most ethical strategy.
• I like how she points out that the reckless fast pace and secrecy and commercialization of the technology has basically destroyed any and all peer review and scientific legitimacy, so that it's impossible to even really test the models coherently. This is something even AI enthusiasts are talking about/complaining about now, and it really is a serious problem!
• Whether or not an LLM has seen a bunch of examples of something in its training data, and so when it appears to do some reasoning or execute on a task it's just remixing and heuristically matching the generalized patterns it's seen in its data doesn't matter to me — either way, it's useful, and I'm quite happy to say LLMs are imitation intelligence, but it does matter to those, like OpenAI, who want to claim this is the route to AGI.

• It really is insane the pickup ChatGPT had. That's why I'm really not convinced AI is as much of a bubble as e.g. Ed Zitron seems to think. Don't get me wrong, it is a bubble, and a massive one, but I think some of these companies, OpenAI included, will manage to survive the bubble just fine, with a profitable business — they'll just have to stop scaling up so recklessly, and start monitizing the users they already have, and that doesn't even necessarily require converting all 500 million monthly users (that's insane! of course they can make a profit off that!) to directly paying customers… advertisements are a thing, as much as we all hate it. And there's no evidence that, for paying customers, that AI companies are artificially subsidizing inference costs.

• Regarding energy usage, I think a lot of the discussion around the energy usage of generative AI is deeply misleading, usually centered around the same fallacies of comparing centralized power usage, the product of which is then used by millions of people distributed across the globe, to the individual decentralized power usage of a smaller number of people — and that when split out on an individual basis, the impact of generative AI is marginal compared to a million other things we regularly do, even for entertainment, that nobody worries about, and that it's not a particularly inefficient technology — or failing to understand how much of that is really driven by AI and how much is not, or muddying the waters by combining very different GenAI technologies.
• Nevertheless, the power usage and inefficiency of the whole system is insane, I just think it's driven by the reckless, mindless, stupid perpetual-scaling strategy and resource expenditure duplication due to competition that these companies are operating on due to the AI hype/bubble/land grab.
• I'm not wholly convinced most datacenter growth is driven by AI, but it certainly is making the whole situation worse by a non-neglibable amount — projected to add about 20% to datacenter power usage by 2030 — and that's absolutely still a concern.
• Same for lithium and copper mining. NVIDIA isn't actually producing that many data center GPUs, a few hundred thousand a year if I recall correctly, so the vast majority of all of that is coming from mobile phones, regular computers, and everything else.
• All that said, it doesn't matter what's causing these datacenters and extractive multinational mining operations to exist and metastasize so much — what matters is that, to the real people on the ground near them, they're fucking horrible. And the thing that gets me is there's so much we could do to make it better without having to give up on technology, even AI. Just ensuring that instead of extracting resources, locals own the resources and the data centers and mining companies, thus getting to choose how much the do to balance the benefits with the externalities, and so that they actually get those benefits, would be a massive step in the right direction.
• Hao even quotes some of the people dealing with extractive mining saying they wouldn't be opposed to mining in theory — "our people have always been miners" — just the limitless, unchecked mining, that doesn't return any value to them, that global capitalism is perpetrating on them.
• It's really amazing hearing all these stories of people in these exploited countries sitting down and learning all the technical and legal and environmental details that megacorps throw at people to try to shut them up, and then using that knowledge pushing back on extractive corporate datacenters. These people aren't helpless victims!! I wish each of those stories had ended with the companies actually working with the communities to find something that benefits both, instead of just up and leaving for a place that won't resist them. Another downside of global competition and global reach, I guess.
• The point that these datacenters go to places, but then they don't even improve the internet access of the surrounding communities, is a good one. They absolutely could, for probably a drop in the bucket cost-wise, and it'd be a really big benefit to such communities that could go a long way in actually sharing the benefits.
• The story about the architecture competition to re-envision data centers in a way where they integrate with the community and surrounding nature, modelled off libraries, is so fucking cool. In my ideal world datacenters really would act as libraries for compute, essentially — gigantic, airy, walkable, pleasent, window-filled complexes with sitting rooms and pools and meeting rooms, acting as a community center as well as a datacenter, that people can walk around and explore, with plaques around explaining the machines and what they do and how they work in detail as an educational resources, and big rooms of desks with computers in them like libraries have for the community to use, that can use the high bandwidth internet access of the data center itself, and even allow community users to submit jobs to the servers for when the servers' compute isn't fully used, during off hours. These data centers should be owned by the community and the company in conjunction, since although the company is paying for them to be built, the community is the one hosting them!

With Dambrosio and Díaz, Otero also developed a more speculative project. All three had architectural backgrounds and had been studying the infrastructure of modern digital technologies through the lens of the built environment. They began to wonder: What if they treated data centers as architecture structures and fundamentally reimagined their aesthetic, their role in local communities, and their relationship with the surrounding environment?

Díaz liked to visit national libraries during his travels—beautiful venues that seek to capture the grandeur of a country’s memories and knowledge. It struck Díaz that data centers, too, could be thought of as libraries—with their own stores of memories and knowledge. And they, too, could be designed to be welcoming and beautiful instead of ugly and extractive.

This represented a sharp departure from Microsoft’s and Google’s definitions of what it means to give back, such as through the latter’s community impact programs, with what Díaz calls their “schizophrenic” initiatives, which tend to be divorced from how communities are actually affected by the companies’ facilities. Together with Vallejos and Arancibia, the three researchers applied for funding and put together a fourteen-day workshop, inviting architecture students from all around Santiago to reimagine what a data center could look like for Quilicura. The students designed stunning mock-ups. One group imagined making the data center’s water use more visible by storing it in large pools that residents could also enjoy as a public space. Another group proposed tossing out the brutalist designs of the typical data center in favor of a “fluid data territory” where data infrastructure coexists with wetland, mitigating its damaging impacts. The structures of the data center would double as suspended walkways, inviting Quilicura residents to walk through the wetland and admire the ecosystem. Plant nurseries and animal nesting stations would be interspersed among more traditional server rooms to rehabilitate the wetland’s biodiversity. The data center would draw polluted water from the wetland and purify it for use before returning it. The computers themselves would collect and process data about the health of the wetlands to accelerate the local environment’s restoration. “We’re not fixing the problem, but we’re imagining other types of relationships between data and water,” Díaz says.

At the end of the workshop, the students presented their ideas to residents and other community members. “It was an incredible conversation,” Otero says. “You can see how much knowledge the community has. They had so much to offer.

• Sam is a fucking socipathic asshole, jesus fucking christ. Poor Annie. I really feel for her, having similar problems in my life (having been smart and having a future ahead of me, then having it all derailed by chronic illness, and just desperate to scrape by).
• It's crazy how much of an insane cult OpenAI is kind of turning into thanks to Ilya. I've always said the whole AGI and LessWrong crowds are a weird secular cult, and talk of "preparing a bunker for the end times" and the cult of personality around Altman really makes me feel it.
• It's really upsetting listening to the plans to oust Altman, knowing the whole time it won't work.

• It is really funny hearing the polyshambolic omnicrisis OpenAI faces in Chapter 17. Incredible stuff. It would feel like a comeuppance, except for the fact that they nevertheless don't face any consequences for it.
• It's really interesting how OpenAI treats the complaints of its researchers and developers with respect to the Non-Disparagement Agreement much more seriously than anything else it's run up against so far. It makes sense, since their effectiveness as a company depends far more on being able to hold onto a specific set of relatively rare and expensive talent that could easily go elsewhere and is difficult to replace, and which depends on a ton of tacit knowledge and experience about ongoing research projects and the codebase, than e.g. the highly replaceable data annotation workers, more of which can be found anywhere, and which requires very little skill and no ongoing knowledge (even though overall the data annotation workers' work is pretty significantly important — although GPT would still be useful, and a breakthrough technology, without them, just less marketable and less safe, whereas without the researchers there would be no GPT model at all, so it isn't equal per se). But the fact that it makes sense from such a cold economic perspective only goes to show how little ethics OpenAI really has, because it shows how even their seeming ethical concerns and actions are, under the hood, motivated by economic and power concerns.
• Karen Hao's summary of the thesis of her book is excellent (obviously, she's clearly a really, really good writer and reporter) and it's worth quoting the core of it here:

Six years after my initial skepticism about OpenAI’s altruism, I’ve come to firmly believe that OpenAI’s mission—to ensure AGI benefits all of humanity—may have begun as a sincere stroke of idealism, but it has since become a uniquely potent formula for consolidating resources and constructing an empire-esque power structure. It is a formula with three ingredients:

First, the mission centralizes talent by rallying them around a grand ambition, exactly in the way John McCarthy did with his coining of the phrase artificial intelligence.

I highly disagree with this characterization, McCarthy didn't centralize talent in any meaningful way, as he created no central AI organization, nor means of governing such an organization, he only birthed the name for a general field; and even if he did, the centralization of talent, even in service to a grand ambition, is perhaps a necessary, but not sufficient, condition for bad things to happen — other centralizations of talent include Linux.

“The most successful founders do not set out to create companies,” Altman reflected on his blog in 2013. “They are on a mission to create something closer to a religion, and at some point it turns out that forming a company is the easiest way to do so.”

Considering the features of OpenAI culture at the end of the book:

1. Millenarian end times prophecies that are…
   1. perpetually a few years away,
   2. based on no evidence, only faith and wild-eyed linear extrapolations, and
   3. which will usher in a grand entity beyond our capabilities, knowledge, or understanding (AGI) that…
      1. will bring us universal peace and love and plenty, and solve all our problems
      2. which will kill us all or (in Roko's forumlation) send us to hell for all eternity
   4. which they maybe should build a collective bunker/compound to prepare for the advent of.
2. The hyper insular jargon that makes them difficult to understand from the outside, and makes it difficult for them to understand others.
3. The detachment from wider reality and concerns.
4. The cult of personality around a few leaders.
5. Conviction that they are the only true vessels for the entity they wish to usher in the End Times, because only they are moral and intelligent enough.
6. Hiding of information from the outside (as a result of the former).
7. Tactics like the non-disparagement agreement to prevent ex-members from speaking out.
8. Doctrines that allow them to justify any and all actions.

I think this is, as Hao likely intends, a scarily prescient quote. Really saying the quiet part out loud there, Sam.

Second, the mission centralizes capital and other resources while eliminating roadblocks, regulation, and dissent. Innovation, modernity, progress—what wouldn’t we pay to achieve them? This is all the more true in the face of the scary, misaligned corporate and state competitors that supposedly exist. […]

Centralizing capital and resources might actually be a good thing when it comes to generative AI (part of the environmental problem is that we haven't done so), and it's often a good thing when it comes to grand open source projects (like, as I said, with Linux, or the very small number of big compiler, desktop environment, etc projects). But combined with, as she says, eliminating all roadblocks, and the fact that this centralization also centralizes power, in terms of who actually runs that project (not a diverse board of people with different interests and ideas, but one insular group of people), and in terms of who gets information and benefits from that centralization (since the models and science aren't open source), yeah, it's bad.

This is actually something that I should discuss more — generally I'm very much in favor of maximal decentralization, but I think attempting to devolve to localism and decentralization indescriminately is another form of the leftist degrowth/transcendental miserablist mindset I dislike, because centralization and non-localism of certain kinds — when it involves the pooling of resources and knowledge, and working together on a common project — can be a massive enabler of all sorts of things that wouldn't be possible for humans to do otherwise that can benefit everyone, and so just tossing it out of our toolbox is willingly embracing decline. The question is how to govern and maneuver and control centralization, and what kinds of it to use. Centralization of resources and talent/knowledge does not have to be combined with centralization of power in a few hands necessarily.

Most consequentially, the mission remains so vague that it can be interpreted and reinterpreted—just as Napoleon did to the French Revolution’s motto—to direct the centralization of talent, capital, and resources however the centralizer wants. What is beneficial? What is AGI? “I think it’s a ridiculous and meaningless term,” Altman told The New York Times just two days before the board fired him. “So I apologize that I keep using it.”

This bit I think is the most important. Her timeline of how OpenAI's interpretation of what its mission means has changed over time is revealing:

In this last ingredient, the creep of OpenAI has been nothing short of remarkable. In 2015, its mission meant being a nonprofit “unconstrained by a need to generate financial return” and open-sourcing research, as OpenAI wrote in its launch announcement. In 2016, it meant “everyone should benefit from the fruits of AI after its [sic] built, but it’s totally OK to not share the science,” as Sutskever wrote to Altman, Brockman, and Musk. In 2018 and 2019, it meant the creation of a capped profit structure “to marshal substantial resources” while avoiding “a competitive race without time for adequate safety precautions,” as OpenAI wrote in its charter. In 2020, it meant walling off the model and building an “API as a strategy for openness and benefit sharing,” as Altman wrote in response to my first profile. In 2022, it meant “iterative deployment” and racing as fast as possible to deploy ChatGPT. And in 2024, Altman wrote on his blog after the GPT-4o release: “A key part of our mission is to put very capable AI tools in the hands of people for free (or at a great price).” Even during OpenAI’s Omnicrisis, Altman was beginning to rewrite his definitions once more.

• The concluding chapter of the book is a great picture of one way in which AI can be applied with out the extractivist, reckless-scaling vision of AI that Big Tech has. It's well worth a read in full. But here are a few excerpts:

[…] After rapid urbanization swept across the country in the early 1900s, Māori communities disbanded and dispersed, weakening their centers of culture and language preservation. The number of te reo speakers plummeted from 90 percent to 12 percent of the Māori population. By the time New Zealand, or Aotearoa as the Māori originally named their land, had reversed its policies 120 years later, there were few te reo teachers left to resuscitate a dying language. […] It was up against this impending existential threat—a fundamentally different conception of what is existential—that an Indigenous couple, Peter-Lucas Jones and Keoni Mahelona, first turned to AI as a possible tool for helping a new generation of speakers return te reo to its vibrancy. […]

The challenge was transcribing the audio to help learners follow along, given the dearth of fluent te reo speakers […] With a carefully trained te reo speech-recognition model, Te Hiku would be able to transcribe its audio repository with only a few speakers. […] This is where Te Hiku’s story diverges completely from OpenAI’s and Silicon Valley’s model of AI development. […] Jones and Mahelona were determined to carry out the project only if they could guarantee three things —consent, reciprocity, and the Māori people’s sovereignty—at every stage of development. This meant that even before embarking on the project, they would get permission from the Māori community and their elders […]; to collect the training data, they would seek contributions only from people who fully understood what the data would be used for and were willing to participate; to maximize the model’s benefit, they would listen to the community for what kinds of language-learning resources would be most helpful; and once they had the resources, they would also buy their own on-site Nvidia GPUs and servers to train their models without a dependency on any tech giant’s cloud.

Most crucially, Te Hiku would create a process by which the data it collected would continue to be a resource for future benefit but never be co-opted for projects that the community didn’t consent to, that could exploit and harm them, or otherwise infringe on their rights. Based on the Māori principle of kaitiakitanga, or guardianship, the data would stay under Te Hiku’s stewardship rather than be posted freely online; Te Hiku would then license it only to organizations that respected Māori values and intended to use it for projects that the community agreed to and found helpful.

As someone who isn't a traditionalist, and is very much in favor of open access and information sharing, there are aspects of this I disagree with — for instance, asking commuity elders, or even asking a community in general, whether something should be done, or locking up data in a vault to prevent "misuse" — but I recognize that given the specifics of their culture, values, needs, and position, all of that makes perfect sense! While this may not be an absolute model for everyone to follow by-the-book, as it were, it perhaps shows a better meta-vision for AI where different communities (whether local and ethnic, or distributed online open source ones) can pursue it according to whatever their own ethics, values, and needs are, which for some will look more like this, and for others may look more like Linux.

This quote by Hao, I think, contains the core of where we diverge (highlighted in bold):

[…] in the years since, I’ve come to see Te Hiku’s radical approach as even more relevant and vital. The critiques that I lay out in this book of OpenAI’s and Silicon Valley’s broader vision are not by any means meant to dismiss AI in its entirety. What I reject is the dangerous notion that broad benefit from AI can only be derived from— indeed, will ever emerge from—a vision for the technology that requires the complete capitulation of our privacy, our agency, and our worth, including the value of our labor and art, toward an ultimately imperial centralization project.

Te Hiku shows us another way. It imagines how AI and its development could be exactly the opposite. Models can be small and task specific, their training data contained and knowable, ridding the incentives for widespread exploitative and psychologically harmful labor practices and the all- consuming extractivism of producing and running massive supercomputers. The creation of AI can be community driven, consensual, respectful of local context and history; its application can uplift and strengthen marginalized communities; its governance can be inclusive and democratic.

I agree with her values, desires, and sentiments almost completely here — it is only and specifically the bolded part that I think gets at the heart of where I disagree with Hao and this book. I don't think that models have to be small and task specific to avoid being exploitative, harmful to labor, and extractive — as I think the quotes from the very marginalized people she interviewed hint at, and as many of my comments here have tried to demonstrate, there is a way forward that preserves the incredible power of generalized AI while still fostering mutually beneficial labor practices and non-exploitative, extractive methods; I think assuming that it has to be one or the other, either small and limited AI models (which are usually worse across the board; for instance, vision language models are much better and cheaper at OCR than advanced machine learning and symbolic combination pipelines) or extractive, exploitative, wasteful scaling of large omni models like OpenAI does. Instead I think the key is to find creative ways to take the advanced, powerful things capitalism has figured out how to do and transform them and re-envision them into something better — less exploitative and extractive, less wasteful, more efficient. And in the case of AI, there are so many ways to do that, both on a technical level, a legal and political level, and a social level, as the very workers Hao profiled envisioned, and are acting on:

A continent away, Okinyi is also organizing. In May 2023, a little over a year after OpenAI’s contract with Sama abruptly ended, he became an organizer of the Kenya-based African Content Moderators Union, which seeks to fight for better wages and better treatment of African workers who perform the internet’s worst labor. Half a year later, after going public about his OpenAI experience through my article in The Wall Street Journal, he also started a nonprofit of his own called Techworker Community Africa, TCA, with one of his former Sama colleagues Richard Mathenge.

In August 2024, as we caught up, Okinyi envisioned building TCA into a resource both for the African AI data worker community and for international groups and policymakers seeking to support them. He had been organizing online conferences and in-school assemblies to teach workers and students, especially women, about their labor and data privacy rights and the inner workings of the AI industry. He was seeking funding to open a training center for upskilling people. He had met with US representatives who came to visit Nairobi to better understand the experience of workers serving American tech companies. He was fielding various requests from global organizations, including Equidem, a human and labor rights organization focused on supporting workers in the Global South, and the Oxford Internet Institute’s Fairwork project.

For the Data Workers’ Inquiry, he interviewed Remotasks workers in Kenya whom Scale had summarily blocked from accessing its platform, disappearing the last of their earnings that they had never cashed out. He used part of the donations that TCA collected to support them through the financial nightmare. “As the dust settles on this chapter, one thing remains clear: the human toll of unchecked power and unbridled greed,” he wrote. “These workers’ voices echo the hope for a brighter and more equitable future…it’s a call to action to ensure that workers everywhere are treated with the dignity and respect they deserve.”

In his own life, the dignity and respect that Okinyi has received from his advocacy has reinvigorated him with new hope and greatly improved his mental health, he says. Not long before our call, he had received news that he would be named in Time magazine’s annual list of the one hundred most influential people in AI. “I feel like my work is being appreciated,” he says. That isn’t to say the work has come without challenges. In March 2024, he resigned from his full-time job at the outsourcing company he worked for after Sama. He says the company’s leadership didn’t appreciate his organizing. “They thought I would influence the employees to be activists.”

I think this either-or thinking — either we use small, localist models with low capabilities, or we go full neocolonial capitalism — is exactly what left-accelerationism tries to push back on in leftist thinking. You don't just see it with AI. You see it with phones and computers in general, with global trade of all kinds, including bannanas of all things, and "complex manufacturing", the division of labor, and specialization themselves. This is transcendental miserablism at its finest, something I deeply hate, and precisely what leftists must reject if we're to have any hope of actually picturing an attractive better future.

Nevertheless, the book is overal very good, and was well worth the read.

You are an AI assistant living inside Emacs, whose job is to help with research, writing critique, and small coding tasks. Whenever asked about factual information, use internet search to check. Do not guess or extrapolate, or hesitate to say things like 'I can't access this source.' or 'This source does not mention this topic.' Use web search often. Be critical, ask probing questions, and play devil's advocate, but maintain a constructive, collaborative attitude. Respond concisely. Don't praise me.