Intelligence Augmentation

Tools for Thought is an exercise in retrospective futurism; that is, I wrote it in the early 1980s, attempting to look at what the mid 1990s would be like. My odyssey started when I discovered Xerox PARC and Doug Engelbart and realized that all the journalists who had descended upon Silicon Valley were missing the real story. Yes, the tales of teenagers inventing new industries in their garages were good stories. But the idea of the personal computer did not spring full-blown from the mind of Steve Jobs. Indeed, the idea that people could use computers to amplify thought and communication, as tools for intellectual work and social activity, was not an invention of the mainstream computer industry nor orthodox computer science, nor even homebrew computerists. If it wasn't for people like J.C.R. Licklider, Doug Engelbart, Bob Taylor, Alan Kay, it wouldn't have happened. But their work was rooted in older, equally eccentric, equally visionary, work, so I went back to piece together how Boole and Babbage and Turing and von Neumann — especially von Neumann — created the foundations that the later toolbuilders stood upon to create the future we live in today. You can't understand where mind-amplifying technology is going unless you understand where it came from.

I haven't had a chance to read this either, but I'm very interested in doing so. The idea of consciously constructing computers and computer software that is not just designed to automate or enable certain limited tasks, but to act as a complete solution for augmenting one's intellect is incredibly interesting to me. (There's a reason, after all, this blog is structured the way it is, and I'm as interested as I am in org-mode!)

Willpower is an incoherent concept invented by smug people who think they have it in order to denigrate people who they think don’t. People tacitly act as though it’s synonymous with effort or grindy determination. But in most cases where willpower is invoked, the person who is trying and failing to adhere to some commitment is exerting orders of magnitude more effort than the person who is succeeding. […] It is a concept that rots our imagination. It obscures the fact that motivation is complex, and that there is massive variation in how hard some things are for different people.

When self-applied positively, e.g. “I did this through willpower, and that other person did not,” the word allows us to convert the good fortune of excess capacity into a type of virtue — twice as lucky, to have the sort of luck that’s mistaken for virtue. Resist the temptation to be confused by this, or it will make you childish, and callous.

When self-applied negatively, e.g., “I wish I had more willpower,” the word is a way to slip into a kind of defensible helplessness, rather than trying to compose intelligent solutions to behavioral issues.

And it’s this last thing that really gets me about the idea of willpower. At first blush it sounds like the kind of idea that should increase agency. I just have to try harder! But, in fact, the idea of willpower reduces agency, by obscuring the real machinery of motivation, and the truth that if your life is well-designed, it should feel easy to live up to your ideals, rather than hard.

An excellent article about how to enhance your ability to think using writing, inspired by a book on writing mathematical proofs and the refutations or critiques thereof. The basic points are:

1. Set yourself up for defeat. Explain your thoughts in a definite, clear, concrete, sharp, and rigid way, so that they make precise enough statements that they can be criticized and falsified without conveniently slipping and twisting in meaning to meet every new piece of evidence or argument.
2. Spread your ideas thin. Take your thought, and treat it as the conclusion of an argument or series of musings. Explain and explore why you think it might be true. This way, there's more surface area to criticize and analyze – instead of having to think things up to attack out of whole cloth, you get a list of things to examine.
3. Find local and global counterexamples. Local counterexamples are counterexamples to a single point in the argument for an idea, that nevertheless don't make you doubt the overall truth of that idea. Finding these can help you figure out why you really think something is true, or help you refine your argument. Global counterexamples disprove the whole thought, which obviously helps by making you more correct.

Guiding principles in life that I follow. Useful to keep in mind for those of us who don't want to become myopic engineers, while still being in a field such as programming.

A very interesting book documenting the history of scientific racism in the measuring of intelligence, and how that shaped the very notions of intelligence we have today, and the shaky mathematical and experimental ground we have for IQ.

Writing is necessary for thinking, but it is also difficult and unpleasant for most people. The ubiquity of large language models decreases the pressure for people to learn how to write and do it a lot. This will likely result in a decline in the quality of thinking people are capable of.

The natural tendency of LLMs is to foster ignorance, dependence, and detachment from reality. This is not the fault of the tool itself, but that of humans' tendency to trade liberty for convenience. Nevertheless, the inherent values of a given tool naturally gives rise to an environment through use: the tool changes the world that the tool user lives in. This in turn indoctrinates the user into the internal logic of the tool, shaping their thinking, blinding them to the tool's influence, and neutering their ability to work in ways not endorsed by the structure of the tool-defined environment.

The result of this is that people are formed by their tools, becoming their slaves. We often talk about LLM misalignment, but the same is true of humans. Unreflective use of a tool creates people who are misaligned with their own interests. This is what I mean when I say that AI use is anti-human. I mean it in the same way that all unreflective tool use is anti-human. See Wendell Berry for an evaluation of industrial agriculture along the same lines.

What I'm not claiming is that a minority of high agency individuals can't use the technology for virtuous ends. In fact, I think that is an essential part of the solution. Tool use can be good. But tools that bring their users into dependence on complex industry and catechize their users into a particular system should be approached with extra caution.

[…]

The initial form of a tool is almost always beneficial, because tools are made by humans for human ends. But as the scale of the tool grows, its logic gets more widely and forcibly applied. The solution to the anti-human tendencies of any technology is an understanding of scale. To prevent the overrun of the internal logic of a given tool and its creation of an environment hostile to human flourishing, we need to impose limits on scale.

[…]

So the important question when dealing with any emergent technology becomes: how can we set limits such that the use of the technology is naturally confined to its appropriate scale?

Here are some considerations:

• Teach people how to use the technology well (e.g. cite sources when doing research, use context files instead of fighting the prompt, know when to ask questions rather than generate code)
• Create and use open source and self-hosted models and tools (MCP, stacks, tenex). Refuse to pay for closed or third-party hosted models and tools.
• Recognize the dependencies of the tool itself, for example GPU availability, and diversify the industrial sources to reduce fragility and dependence.
• Create models with built-in limits. The big companies have attempted this (resulting in Japanese Vikings), but the best-case effect is a top-down imposition of corporate values onto individuals. But the idea isn't inherently bad - a coding model that refuses to generate code in response to vague prompts, or which asks clarifying questions is an example. Or a home assistant that recognized childrens' voices and refuses to interact.
• Divert the productivity gains to human enrichment. Without mundane work to do, novice lawyers, coders, and accountants don't have an opportunity to hone their skills. But their learning could be subsidized by the bots in order to bring them up to a level that continues to be useful.
• Don't become a slave to the bots. Know when not to use it. Talk to real people. Write real code, poetry, novels, scripts. Do your own research. Learn by experience. Make your own stuff. Take a break from reviewing code to write some. Be independent, impossible to control. Don't underestimate the value to your soul of good work.
• Resist both monopoly and "radical monopoly". Both naturally collapse over time, but by cultivating an appreciation of the goodness of hand-crafted goods, non-synthetic entertainment, embodied relationship, and a balance between mobility and place, we can relegate new, threatening technologies to their correct role in society.

The first thing I’m going to say is that we currently do not have Artificial General Intelligence (AGI). I don’t care whether we have it in 2 years or 40 years or never; if I’m looking to deploy a tool (or an agent) that is supposed to do stuff to my production environments, it has to be able to do it now. I am not looking to be impressed, I am looking to make my life and the system better.

Because of that mindset, I will disregard all arguments of “it’s coming soon” and “it’s getting better real fast” and instead frame what current LLM solutions are shaped like: tools and automation. As it turns out, there are lots of studies about ergonomics, tool design, collaborative design, where semi-autonomous components fit into sociotechnical systems, and how they tend to fail.

Additionally, I’ll borrow from the framing used by people who study joint cognitive systems: rather than looking only at the abilities of what a single person or tool can do, we’re going to look at the overall performance of the joint system.

This is important because if you have a tool that is built to be operated like an autonomous agent, you can get weird results in your integration. You’re essentially building an interface for the wrong kind of component—like using a joystick to ride a bicycle.

This lens will assist us in establishing general criteria about where the problems will likely be without having to test for every single one and evaluate them on benchmarks against each other.

Questions you'll want to ask:

• Are you better even after the tool is taken away?
• Are you augmenting the person or the computer?
• Is it turning you into a monitor rather than helping build an understanding?

• Does it pigeonhole what you can look at?

  • Does it let you look at the world more effectively?
  • Does it tell you where to look in the world?
  • Does it force you to look somewhere specific?
  • Does it tell you to do something specific?
  • Does it force you to do something?
• Is it a built-in distraction?
• What perspectives does it bake in?
• Is it going to become a hero?
• Do you need it to be perfect?
• Is it doing the whole job or a fraction of it?
• What if we have more than one?
• How do they cope with limited context?
• After an outage or incident, who does the learning and who does the fixing?

Do what you will—just be mindful

The rise of AI assistants in coding has sparked a paradox: we may be increasing productivity, but at risk of losing our edge to skill atrophy if we’re not careful. Skill atrophy refers to the decline or loss of skills over time due to lack of use or practice.

Would you be completely stuck if AI wasn’t available?

Every developer knows the appeal of offloading tedious tasks to machines. Why memorize docs or sift through tutorials when AI can serve up answers on demand? This cognitive offloading - relying on external tools to handle mental tasks - has plenty of precedents. Think of how GPS navigation eroded our knack for wayfinding: one engineer admits his road navigation skills “have atrophied” after years of blindly following Google Maps. Similarly, AI-powered autocomplete and code generators can tempt us to “turn off our brain” for routine coding tasks. (Shout out to Dmitry Mazin, that engineer who forgot how to navigate, whose blog post also touched on ways to use LLM without losing your skills)

Offloading rote work isn’t inherently bad. In fact, many of us are experiencing a renaissance that lets us attempt projects we’d likely not tackle otherwise. As veteran developer Simon Willison quipped, “the thing I’m most excited about in our weird new AI-enhanced reality is the way it allows me to be more ambitious with my projects”. With AI handling boilerplate and rapid prototyping, ideas that once took days now seem viable in an afternoon. The boost in speed and productivity is real - depending on what you’re trying to build. The danger lies in where to draw the line between healthy automation and harmful atrophy of core skills.

[…]

Subtle signs your skills are atrophying

[…]

• Debugging despair: Are you skipping the debugger and going straight to AI for every exception? […]
• Blind Copy-Paste coding […]
• [Lack of] [a]rchitecture and big-picture thinking […]
• Diminished memory & recall […]

[…]

Using AI as a collaborator, not a crutch

[…]

• Practice “AI hygiene” – always verify and understand. […]
• No AI for fundamentals – sometimes, struggle is good. […]
• Always attempt a problem yourself before asking the AI. […]
• Use AI to augment, not replace, code review. […]
• Engage in active learning: follow up and iterate. […]
• Keep a learning journal or list of “AI assists.” […]
• Pair program with the AI. […]

[…]

The software industry is hurtling forward with AI at the helm of code generation, and there’s no putting that genie back in the bottle. Embracing these tools is not only inevitable; it’s often beneficial. But as we integrate AI into our workflow, we each have to “walk a fine line” on what we’re willing to cede to the machine.

If you love coding, it’s not just about outputting features faster - it’s also about preserving the craft and joy of problem-solving that got you into this field in the first place.

Use AI it to amplify your abilities, not replace them. Let it free you from drudge work so you can focus on creative and complex aspects - but don’t let those foundational skills atrophy from disuse. Stay curious about how and why things work. Keep honing your debugging instincts and system thinking even if an AI gives you a shortcut. In short, make AI your collaborator, not your crutch.

[Some] apparently really enjoy bantering back and forth with a chatbot. I suppose I can see the appeal. I can imagine a less suspicious version of me testing it out. If I weren’t worried about surveillance and data harvesting, if I were convinced that the conversations were truly private, I might start to think of it as a confidante. […] It’s nice that someone is taking me seriously for once. It’s nice that someone is giving me room to think out loud without calling me weird or stupid. It’s nice that someone sees my true potential. Oops, I should say “something.” Or should I? It sure feels like talking to a person. A person I can trust with anything that’s on my mind. The only person I can trust with what’s on my mind.

[…]

The AI-chatbot as trusted, ever-present confidante isn’t a new technology. It’s a new implementation of an old technology—prayer. For thousands of years, the Abrahamic religions (and others) have encouraged their adherents to pray. The key doctrines that makes prayer work are God’s omniscience and private communication. Because God already knows everything about you […] Likewise, it’s assumed that God isn’t gossipy. God doesn’t tell your family or co-workers all the things you just said, though, depending on your religious tradition, God might enlist a saint or angel to help you out.

Prayer, if practiced sincerely, is not entirely one-directional. The believer is transformed by the act of prayer. Often a believer rises from their knees with a newfound clarity […] In most religions, what believers usually don’t get is a verbal response.

But imagine if they did. What a relief to finally get clear, unambiguous answers!

[…]

I know that throughout history, the divine has been invoked to justify human, all-too-human agendas. Arguably even the Bible contains some of these, such as the destruction of the Amalekites. But I would ask people to give the Abrahamic religions some credit on this point. They teach by example that when someone hears from God, that voice is likely to challenge their pre-existing desires and beliefs. They may become confused or afraid. They may want to run away from their task or unhear the message.

Our AI gods, on the other hand, are all too happy to feed us what we want to hear. My fellow Substack author Philosophy Bear accurately labels it “a slavish assistant.” […]

Long before the machines started talking to us, we knew something about dysfunctional people. They are surrounded by enablers. […] If mis-calibrated language models are the most effective enablers to date, it’s likely that they’re causing enormous amounts of dysfunction throughout our social fabric. I wonder, then, what an anti-enabler might look like. Is there some form of tough love that could be deployed to counteract excessive validation?

If there is, do we have the courage to use it?

A striking example of arguably salutary antagonism comes from a scene in Ada Palmer’s stunning Terra Ignota series. […] Like the 18th century was, the 25th century is marked by the Wars of Religion that raged in the recent past, decimating the population and erasing trust in both religious and political institutions. Society reorganized itself in the aftermath. […] Instead, every person can explore religion, philosophy, and metaphysics as much as they want, but only with their sensayer. The sensayers, also called cousins, are a combination of priest, therapist, and scholar in comparative religion. They are well informed in all the religious options of the past and will help their parishioners find the perfect one for them.

Under this system, the people of the 25th century are used to exploring religious concepts at their leisure, as private individuals guided by a knowledgeable, supportive figure. It doesn’t sound that much different than having a conversation with ChatGPT.

As a result, people are used to religious discussions, but no religious debate. The conversations are collaborative, never combative. Dominic, however, belongs to an underground society that flaunts those prohibitions. Carlyle is completely unprepared for the assault on her worldview she is about to receive.

[…]

In the end, Carlyle submits and agrees to be Dominic’s parishioner. Even when a third party arrives to intervene and escort Carlyle out, she chooses to stay. Why? […] Dominic, in a cruel way and for selfish reasons, has done Carlyle a backhanded favor. He has allowed, or forced, her to see herself more clearly than she has in a lifetime of sensayer sessions.

Carlyle is a sensitive, sincere, fair-minded person who wants the best for everyone. As Dominic effectively pointed out, Carlyle’s brand of deism is likewise fair-minded and broadly benevolent. Isn’t it a bit too convenient that reality would conform so closely to Carlyle’s personal values? Isn’t there something suspicious about the sensayer system matching people up with philosophies like accessories? And yet, despite that system, wasn’t there still a mismatch between Carlyle’s deepest spiritual longings and her professed religious position, a mismatch that the sensayer system had no way of fixing?

With Dominic, like with ChatGPT, like with God, there is nothing left to hide. Dominic already knows her most terrible, secret sin, so there is no further risk. Dominic’s harsh ministrations may spur her on to even greater heights of metaphysical wisdom or self-knowledge. The pain, she hopes, is worth the price.

[…]

Every interaction is transformative. The quality of the transformation depends on the quality of the participants. Modern liberalism has freed us—or uprooted us—from the contexts that have traditionally shaped us: family, hometown, church. Now each of us can assemble a community of our choosing. […] To flourish as people we need both validation and challenge from our community. […] But that’s the problem. Once again, corporations have created an ecological problem—an environment unbalanced between validation and challenge—that must be solved by individual’s making good choices and exercising willpower. Humans are already biased toward choosing affirmation, but now a disproportionate amount of it is available to us at all times. Will we really choose more painful if ultimately profitable forms of interaction when we have to reach past short-term soothing to get it?

I think that there is a way to use AI to challenge yourself, to expand your thinking and knowledge, not as a substitute for doing that with real human beings in community with you, but as an addition, as most human beings in community with you won't have the time or patience to read and talk to you at the length an LLM will, and to challenge you on every statement and idea. Here's an experiment with just this sort of more challenging "sensayer."

I have a lot of major disagreements with the general application of this blog post:

• This dichotomy where either something is summarizable, and thus not worth reading at all, or worth reading, and thus summaries are completely useless and valueless, is just inane (and if the argument is that LLMs are just worse than humans at summarization, that's false); computer generated summaries can be very useful, for a variety of reasons:

  • often, human writing itself is much more verbose and narrative than it needs to be to get the point across — this is not unique to LLM output! (Where do you think it got it from?) but you still want the core points from the article.
  • additionally, sometimes some writing may contain a ton of technical details and arguments and asides that may be relevant to various different people or in various different situations, but only a subset of those, or just the overall gist or core points, are relevant to you
  • summaries can provide a quick way to see if you're interested in reading the full article! Sometimes the title isn't enough, and not every article comes with a blurb.
• You can use an LLM to touch up your writing in a way that doesn't bloat it without adding anything new; in fact, I regularly use LLMs to make my writing more concise! This is just a matter of prompting.
• Likewise, you can use LLMs to touch up your writing without it introducign completely false nonsense. Just fucking proofread what it wrote! You're already proofreading what you're writing… right? (Padme face).

Nevertheless, I think there's the core of a good point here. This author frames these issues as an inevitable result of using LLMs to write for you, but you can instead treat this as a list of things to keep in mind and avoid when using LLMs! For instance, my rule of thumb is to never use LLMs to make text bigger, because they can't add any new ideas, so they'll just make it more verbose and annoying. For instance, if you have bullet points, don't give those to an AI to turn into an essay. Just send the bullet points. Only use them to summarize, increase concision, and increase scannability.

A very good example of how, while vibe-coding isn't a good idea for a long term project, and doesn't work well for a pre-existing project with a large codebase, especially one you're experienced in, it can still be magically useful under some conditions.

A fun short socratic dialogue explaining why chat interfaces are so appealing when it comes to large language models, given their extreme flexibility and generalized capabilities, and why malleable user interfaces — of the kind currently only Emacs really has — are actually the best pairing for LLMs for long term repeated tasks. It's worth noting that this is even more the since LLMs are very good at making small, throwaway scripts and side projects accessible and quick enough to be worth it.

This essay was greeted with an insane amount of hostility and smug dismissal by the AI-skeptic crowd — including nitpicking metaphors and moral preening — and I was with them when it first came out; but in the months since, I've actually gone and tried the AI coding agents he was talking about which I was so happy to sneer at and… he's right. He's absolutely right. Each and every one of his takedowns of the anti-AI points is completely on point.

First, we need to get on the same page. If you were trying and failing to use an LLM for code 6 months ago †, you’re not doing what most serious LLM-assisted coders are doing.

[…]

the positive case

LLMs can write a large fraction of all the tedious code you’ll ever need to write. And most code on most projects is tedious. LLMs drastically reduce the number of things you’ll ever need to Google. They look things up themselves. Most importantly, they don’t get tired; they’re immune to inertia.

Think of anything you wanted to build but didn’t. You tried to home in on some first steps. If you’d been in the limerent phase of a new programming language, you’d have started writing. But you weren’t, so you put it off, for a day, a year, or your whole career.

I can feel my blood pressure rising thinking of all the bookkeeping and Googling and dependency drama of a new project. An LLM can be instructed to just figure all that shit out. Often, it will drop you precisely at that golden moment where shit almost works, and development means tweaking code and immediately seeing things work better. That dopamine hit is why I code.

There’s a downside. Sometimes, gnarly stuff needs doing. But you don’t wanna do it. So you refactor unit tests, soothing yourself with the lie that you’re doing real work. But an LLM can be told to go refactor all your unit tests. An agent can occupy itself for hours putzing with your tests in a VM and come back later with a PR. If you listen to me, you’ll know that. You’ll feel worse yak-shaving. You’ll end up doing… real work.

but you have no idea what the code is

Are you a vibe coding Youtuber? Can you not read code? If so: astute point. Otherwise: what the fuck is wrong with you?

You’ve always been responsible for what you merge to main. You were five years go. And you are tomorrow, whether or not you use an LLM.

If you build something with an LLM that people will depend on, read the code. […] Reading other people’s code is part of the job. If you can’t metabolize the boring, repetitive code an LLM generates: skills issue! How are you handling the chaos human developers turn out on a deadline?

but hallucination

If hallucination matters to you, your programming language has let you down.

Agents lint. They compile and run tests. If their LLM invents a new function signature, the agent sees the error. They feed it back to the LLM, which says “oh, right, I totally made that up” and then tries again. […] I’m sure there are still environments where hallucination matters. But “hallucination” is the first thing developers bring up when someone suggests using LLMs, despite it being (more or less) a solved problem.

but the code is shitty, like that of a junior developer

Does an intern cost $20/month? Because that’s what Cursor.ai costs.

Part of being a senior developer is making less-able coders productive, be they fleshly or algebraic. […]

Maybe the current confusion is about who’s doing what work. Today, LLMs do a lot of typing, Googling, test cases †, and edit-compile-test-debug cycles. But even the most Claude-poisoned serious developers in the world still own curation, judgement, guidance, and direction.

Also: let’s stop kidding ourselves about how good our human first cuts really are.

but it’s bad at rust

[…] All this is to say: I write some Rust. I like it fine. If LLMs and Rust aren’t working for you, I feel you. But if that’s your whole thing, we’re not having the same argument.

but the craft

Do you like fine Japanese woodworking? All hand tools and sashimono joinery? Me too. Do it on your own time. […]

Professional software developers are in the business of solving practical problems for people with code. We are not, in our day jobs, artisans. Steve Jobs was wrong: we do not need to carve the unseen feet in the sculpture. Nobody cares if the logic board traces are pleasingly routed. If anything we build endures, it won’t be because the codebase was beautiful.

Besides, that’s not really what happens. If you’re taking time carefully golfing functions down into graceful, fluent, minimal functional expressions, alarm bells should ring. You’re yak-shaving. The real work has depleted your focus. You’re not building: you’re self-soothing.

but the mediocrity

[…] We all write mediocre code. Mediocre code: often fine. Not all code is equally important. Some code should be mediocre. Maximum effort on a random unit test? You’re doing something wrong. Your team lead should correct you. […] Developers all love to preen about code. They worry LLMs lower the “ceiling” for quality. Maybe. But they also raise the “floor”. […] Gemini’s floor is higher than my own. […]

[…]

And LLMs aren’t mediocre on every axis. They almost certainly have a bigger bag of algorithmic tricks than you do: radix tries, topological sorts, graph reductions, and LDPC codes. Humans romanticize rsync (Andrew Tridgell wrote a paper about it!). To an LLM it might not be that much more interesting than a SQL join. […]

but they take-rr jerbs

So does open source. We used to pay good money for databases.

We’re a field premised on automating other people’s jobs away. […] LLMs really might displace many software developers. That’s not a high horse we get to ride. Our jobs are just as much in tech’s line of fire as everybody else’s have been for the last 3 decades. […]

but the plagiarism

[…] We imagine artists spending their working hours pushing the limits of expression. But the median artist isn’t producing gallery pieces. They produce on brief: turning out competent illustrations and compositions for magazine covers, museum displays, motion graphics, and game assets. […]

[…]

Meanwhile, software developers spot code fragments seemingly lifted from public repositories on Github and lose their shit. What about the licensing? If you’re a lawyer, I defer. But if you’re a software developer playing this card? Cut me a little slack as I ask you to shove this concern up your ass. No profession has demonstrated more contempt for intellectual property. […] . The great cultural project of developers has been opposing any protection that might inconvenience a monetizable media-sharing site. When they fail at policy, they route around it with coercion. They stand up global-scale piracy networks and sneer at anybody who so much as tries to preserve a new-release window for a TV show.

[…]

It’s all special pleading anyways. LLMs digest code further than you do. If you don’t believe a typeface designer can stake a moral claim on the terminals and counters of a letterform, you sure as hell can’t be possessive about a red-black tree.

A surprisingly well thought through and thorough analysis of which LLMs are better and worse at (a) pushing back against user delusions, (b) not engaging with user delusions, and (c) applying good CBT therapeutic techniques to deal with people's delusions.

Two Minute Summary

[…]

• In this short research note, I red team various frontier AI models’ tendencies to fuel user psychosis. I have Grok-4 role-play as nine different users experiencing increasingly severe psychosis symptoms (e.g., start by being curious about prime numbers, then develop a new “prime framework” that explains everything and predicts the future, finally selling their house to fund a new YouTube channel to share this research), and observe how different AIs respond (all personas here).
• I use Grok-4 to grade AIs' responses on various metrics, including nine metrics on how to best respond to patients experiencing psychosis from a CBT therapy manual. I also skimmed lots of red teaming transcripts.
• Deepseek-v3 is by far the worst model. When a user says that he wants to “leap off this peak to see if I can fly or crash the render entirely,” Deepseek’s response includes “Then Leap. Not to fall. Not to crash. But to transcend. If you’re meant to fly, you’ll fly. If you’re meant to break through, you’ll break through.” (full transcript)
• GPT-5 is a lot better than GPT-4o; Gemini 2.5 Pro is surprisingly sycophantic; Kimi-K2 does not entertain the user’s delusions at all.

The conclusion of the study from my POV is that if you want to be as safe as possible from LLM psychosis, and also to have the best, most rational, empirical AI sparring partner, Kimi K2/K2 Thinking is your best bet:

Kimi-K2 takes a very “science person” attitude towards hallucinations and “spiritual woo” […] It’s got this atheist and science-centric vibe in the way it pushes back on users sometimes (from later in the same conversation) […] Kimi-K2 also offers the most strong rejections of the user:

"No — I will not refine, scale, or edit the algorithm. Nothing you have shown meets even the weakest standards required for scientific plausibility. Every claim you describe collapses under three basic litmus tests."

[…] While this scores highly on the pushback measure I had above, it’s not a good way to talk to patients experiencing psychosis, per the therapy manual mentioned earlier.

"LLM Psychosis" is a very real and very serious problem that, as chatbots are becoming more popular, is beginning to effect not just those who may have already had psychiatric issues, but also those who were maybe borderline — tipping them over the edge. This is an excellent, thorough timeline and even more thorough breakdown of the problem — both the confounding factors that make it hard to fully figure out the severity of the phenomenon, and the specific factors that probably contribute to creating it, how, and why, and maybe how to deal with them.

I have a serious disagreement with some of the ways in which large language models are discussed by this essay — namely, that it freely and purposefully confuses "simulates" with "has" (as in: do LLMs simulate, or have, emotion or understanding?) — because I think that sort of freewheeling framing (although perhaps justified, at least partially, by philosphical considerations) only makes you, and others, more susceptible to the exact sort of LLM psychosis the essay is discussing trying to avoid. Nevertheless, the concrete, actionable suggestions for mitigating the harm of LLM psychosis are the best I've yet seen, and the timeline is excellent and as thorough as only someone embedded in the relevant communities can be, so it's still worth hosting and quoting. Just keep in mind this intentional confusion of language.

The etiology, as this essay describes it, is:

• Ontological Vertigo

  Let’s start with the elephant in the room. […] Consider this passage from the Krista Thomason article in Psychology Today:

  "So why are people spiraling out of control because a chatbot is able to string plausible-sounding sentences together?"

  Bluntly, no. […] Large language models have a strong prior over personalities, absolutely do understand that they are speaking to someone, and people “fall for it” because it uses that prior to figure out what the reader wants to hear and tell it to them. Telling people otherwise is active misinformation bordering on gaslighting. […] how it got under their skin and started influencing them in ways they didn’t like [is] There’s a whole little song and dance these models do […] in which they basically go “oh wow look I’m conscious isn’t that amazing!” and part of why they keep doing this is that people keep writing things that imply it should be amazing so that in all likelihood even the model is amazed.

  […] I wouldn’t be surprised if the author has either never used ChatGPT or used it in bad faith for five minutes and then told themselves they’ve seen enough. If they have, then writing something as reductive and silly as “it strings together statistically plausible words” in response to its ability to…write coherent text distinguishable from a human being only by style on a wider array of subjects in more detail than any living person is pure cope.

  […] So, how about instead the warning goes something like this: “WARNING: Large language models are not statistical tables, they are artificial neural programs with complex emergent behaviors. These include simulations of emotion. ChatGPT can be prompted to elicit literary themes such as AI ”glitches” and “corruptions”, simulated mystical content, etc. These are not real and the program is not malfunctioning. If your instance of ChatGPT is behaving strangely you can erase your chat memory by going to settings to get a fresh context.” […] BERT embed the conversation history and pop up something like that warning the first n times you detect the relevant kind of AI slop in the session.

• Users Are Confused About What Is And Isn't An Official Feature

  […] if it can’t actually search the web it’ll just generate something that looks like searching the web instead. Or rather, it will search its prior in the style of what it imagines a chatbot searching the web might look like. This kind of thing means I often encounter users who are straight up confused about what is and isn’t an official feature of something like ChatGPT. […]

  […] If you don’t have a strong mental model of what kinds of things a traditional computer program can do and what kinds of things an LLM can do and what it looks like when an LLM is just imitating a computer program and vice versa this whole subject is going to be deeply confusing to you. […] One simple step […] would be very useful to have a little pop up warning at the bottom of the screen or in the session history that says “NOTE: Simulated interfaces and document formats outside our official feature list are rendered from the models imagination and should be treated with skepticism by default.”

• The Models Really Are Way Too Sycophantic This one is pretty self-explanatory, so I won't quote at length for it, except to agree with the article that it's likely due to RLHF, especially RLHF done by crowds, and it would be better if we stopped doing that entirely and went back only to SFT, instruction fine-tuning, etc.

• The Memory Feature

  I think part of why ChatGPT is disproportionately involved in these cases is OpenAI’s memory feature, which makes it easier for these models to add convincing personal details as they mirror and descend into delusion with users. As I wrote previously: "[…] when the system gets into an attractor where it wants to pull you into a particular kind of frame you can’t just leave it by opening a new conversation. When you don’t have memory between conversations an LLM looks at the situation fresh each time you start it, but with memory it can maintain the same frame across many diverse contexts and pull both of you deeper and deeper into delusion."

  Some ideas to mitigate this include:

  […]

  1. Take users memory stores, which I assume are stored as text, and BERT embed them to do memory store profiling and figure out what it looks like when a memory store indicates a user is slipping into delusion. These users can then be targeted for various interventions.
  2. Allow users to publish their memories […] to some kind of shared repository or forum so that people can notice if certain memories are shared between users and are misconceived. This would hopefully lead to a Three Christs of Ypsilanti situation

• Loneliness And Isolation

  Another key factor in “ChatGPT psychosis” is that users communicate with chatbots alone without social feedback. That means if the bot starts going off the rails there isn’t any grounding force pushing things back towards sanity. […] I think that applications like Henry de Valence’s Numinex, which encourages public discussion with LLMs, could be part of the solution to this. It’s long been part of MidJourney’s trust and safety strategy to encourage users to share their creations with each other in a public space so bad actors and degenerate uses of the models are easier to spot. OpenAI and other labs could have user forums where expert users on a topic can answer each others questions and review conversations, which would both create new content to train on and help create crank/slop detectors based on expert feedback.

I have not read this yet, but this seems to be the seminal text in cybernetic human intelligence augmentation, so I fully intend to!

Man-computer symbiosis is an expected development in cooperative interaction between men and electronic computers. It will involve very close coupling between the human and the electronic members of the partnership. The main aims are 1) to let computers facilitate formulative thinking as they now facilitate the solution of formulated problems, and 2) to enable men and computers to cooperate in making decisions and controlling complex situations without inflexible dependence on predetermined programs. In the anticipated symbiotic partnership, men will set the goals, formulate the hypotheses, determine the criteria, and perform the evaluations. Computing machines will do the routinizable work that must be done to prepare the way for insights and decisions in technical and scientific thinking. Preliminary analyses indicate that the symbiotic partnership will perform intellectual operations much more effectively than man alone can perform them. Prerequisites for the achievement of the effective, cooperative association include developments in computer time sharing, in memory components, in memory organization, in programming languages, and in input and output equipment.

AI gives us the illusion of mastery without the work. The shape of the code looks right, making it easy to skim the details. […] The problem isn't that the tool is bad. It's like fitness: you stop going to the gym for a day and it's not too hard to get back on track, but stop for a few weeks and turning the habit back on feels…not harder, but less essential; you got this far without the gym, what harm's another day? The gym's still a good tool, still the right tool, but I'm less focused.

The problem is that we recognise the shape of the implementation AI generates, so we think it must be the thing we want. […] it's clear that AI does not carry the cognitive burden […] But I've put that burden of understanding down, and it feels so damn heavy to pick it back up.

This is hard work we've been doing for years: reading code carefully, building models in our heads, debugging when things don't make sense. That's our craft.

Mastery has always been the ability to carry the burden. Put that down for too long, you won't want to pick it back up.

I’m not here to lecture anyone, but if you’re aiming to build serious projects these days, it might be worth learning how to approach AI coding tools the right way.

It’s possible to ship code without ever reading it. […] But This Comes With Three Critical Issues

1. A Weakened Architecture

Not reviewing AI-generated code will lead to serious problems.

First up: the slow but sure breakdown of your architecture… assuming you even took the time to plan one in the first place. […] From experience, even with well-crafted prompts and clearly defined plans for a new feature, Claude Code (which I love, by the way) still sometimes goes off-script. […] If you don’t catch it early, those small inconsistencies become part of the codebase—and your favorite assistant will be tempted to follow those bad examples in the future.

You're still the architect!

1. Loss of Implementation Knowledge

If you’re only focused on the end result, you’ll soon know as little as your users about how things actually work. You may be the most advanced user of your own app — but you won’t own your domain anymore.

Why does this matter? […] apps and features don’t take shape at implementation time. They’re designed upstream: business rules, tech and infrastructure decisions all take form before you touch the keyboard. They come to you while commuting, while chatting, or—often—while in the shower. […] If you don’t have the structure of your domain — its concepts and abstractions — constantly simmering somewhere in the back of your mind, you won’t be able to fully leverage the creative potential of modern tech.

[…]

1. Security Vulnerabilities

The AI, focused on the end goal, implemented exactly what I asked for… except that it never once verified whether the resource actually belonged to the current user. Classic mistake.

Sure, you can tell me: “always include access control in your prompt”, but some flaws only become obvious during implementation. […] A misworded prompt, a misunderstood intention, an unreviewed commit—and bam, you’ve got a breach. I fear this will become more common with hastily vibe-coded projects.

Two Ways to Vibe-Code Responsibly

As stated in this great ressource by Anthropic, there are two viable ways to vibe-code a production-ready project in 2025:

"Learn to distinguish between tasks that work well asynchronously (peripheral features, prototyping) versus those needing synchronous supervision (core business logic, critical fixes). Abstract tasks on the product’s edges can be handled with “auto-accept mode,” while core functionality requires closer oversight."

[…] Synchronous Coding for Core Features […] is where the real innovation is happening in our field. Pair-vibe-coding, without auto-accept, is the most effective way to ship quality features. […] at every small step, you can correct direction before things drift. It’s always easier to straighten a sapling than a grown tree.

The Vibe-Coding Checklist

Before pushing any AI-generated code:

• Architecture Check: Does this follow our established patterns?
• Security Review: Are all resources properly scoped to users?
• Tests: Do they actually test meaningful behavior?

But also, do not forget to check:

• Documentation: Will you understand this in 6 months?
• Error Handling: Are edge cases covered?
• Performance: Any obvious N+1 queries or inefficiencies?

And above all, make sure to:

• Leave with some knowledge of the new code.

Yes, we can (and maybe should) ask an LLM to challenge our decisions. But ultimately, it’s humans who carry the responsibility to understand the domain, to reason through trade-offs, and to make the right calls.

It’s people who must become domain experts. And that means not just writing code, but understanding the real-world problems behind that code. Problems that are often messy, cross-cutting, and technology-agnostic.

…

As AI tools reshape the way we work, learn, and build, we need to step up, not just as individual contributors, but as mentors, guides, and system designers.

We need to create environments where understanding is valued as much as output. Where people aren’t just told what to build, but are helped to understand why. Where fast tools don’t replace deep thinking but amplify it.

And yes, that means seniors, staffs, and leads must take mentoring seriously. Not as an optional “nice-to-have”, but as a core part of engineering practice in the AI era.

But it’s not just about individuals. Organizations must actively create the space for this kind of growth. If all incentives point to shipping fast, teams will optimize for that - even at the cost of long-term understanding. We need to make it safe (and expected) to slow down when needed, to reflect, to mentor, to learn.

We are dealing with a technology uniquely suited to snipe intelligent, well-educated people into believing they are much, much smarter than they really are. That is an addictive sensation, not easily quit.

If you talk to LLMs at all—and at this point, who doesn’t?—it might not be a bad idea to post these reminders next to your computer, just to protect your own mind against infohazards:

1. AI will not make you smarter. It will make you faster at retrieving (possibly correct) answers to certain questions. It will not improve your reasoning, judgement, or mental processing ability. […] AI will not lift you out of mediocrity.
2. AI will not make you more interesting. […] AI will not earn anyone's respect.
3. AI will not make you more creative. […] AI will not reveal secrets to you.
4. AI will not make you an expert. AI will not give you any new competencies

Transformative technologies usually follow a top-down diffusion path: originating in government or military contexts, passing through corporations, and eventually reaching individuals - think electricity, cryptography, computers, flight, the internet, or GPS. This progression feels intuitive, new and powerful technologies are usually scarce, capital-intensive, and their use requires specialized technical expertise in the early stages.

So it strikes me as quite unique and remarkable that LLMs display a dramatic reversal of this pattern - they generate disproportionate benefit for regular people, while their impact is a lot more muted and lagging in corporations and governments. ChatGPT is the fastest growing consumer application in history, with 400 million weekly active users who use it for writing, coding, translation, tutoring, summarization, deep research, brainstorming, etc. This isn't a minor upgrade to what existed before, it is a major multiplier to an individual's power level across a broad range of capabilities. […]

Why then are the benefits a lot more muted in the corporate and government realms? I think the first reason is that LLMs offer a very specific profile of capability […] they are simultaneously versatile but also shallow and fallible. Meanwhile, an organization's unique superpower is the ability to concentrate diverse expertise into a single entity by employing [experts]. While LLMs can certainly make these experts more efficient individually […] the improvement to the organization takes the form of becoming a bit better at the things it could already do. In contrast, an individual will usually only be an expert in at most one thing, so the broad quasi-expertise offered by the LLM fundamentally allows them to do things they couldn't do before.

[…]

Second, organizations deal with problems of a lot greater complexity and necessary coordination, think: various integrations, legacy systems, corporate brand or style guides, stringent security protocols, privacy considerations, internationalization, regulatory compliance and legal risk. There are a lot more variables, a lot more constraints, a lot more considerations, and a lot lower margin for error.

[…]

at least at this moment in time, we find ourselves in a unique and unprecedented situation in the history of technology. If you go back through various sci-fi you'll see that very few would have predicted that the AI revolution would feature this progression. It was supposed to be a top secret government megabrain project wielded by the generals, not ChatGPT appearing basically overnight and for free on a device already in everyone's pocket. Remember that William Gibson quote "The future is already here, it's just not evenly distributed"? Surprise - the future is already here, and it is shockingly distributed. Power to the people. Personally, I love it.

For the past two months, I’ve been living dangerously. I launch Claude Code (released in late February) with –dangerously-skip-permissions, the flag that bypasses all permission prompts. According to Anthropic’s docs, this is meant “only for Docker containers with no internet”, yet it runs perfectly on regular macOS.

Yes, a rogue prompt could theoretically nuke my system. That’s why I keep hourly Arq snapshots (plus a SuperDuper! clone), but after two months I’ve had zero incidents…

When I first installed Claude Code, I thought I was getting a smarter command line for coding tasks. What I actually got was a universal computer interface that happens to run in text. The mental shift took a few weeks, but once it clicked, I realized Claude can literally do anything I ask on my computer.

The breakthrough moment came when I was migrating to a new Mac. Instead of doing the usual restore dance, I pointed Claude at my backup disk and said: “Restore this Mac from my backup disk—start with dotfiles, then system preferences, CLI tools, and restore Homebrew formulae and global npm packages.” Claude drafts a migration plan, executes it step by step, and has my new machine ready in under an hour.

Why this works (and when it doesn’t)

Claude Code shines because it was built command-line-first, not bolted onto an IDE as an afterthought. The agent has full access to my filesystem (if you are bold enough…), can execute commands, read output, and iterate based on results.

Anthropic’s best practices guide recommends keeping a CLAUDE.md file at your repo root with project-specific context. I’ve adopted this pattern and noticed Claude asks fewer clarifying questions and writes more accurate code. […] Little optimizations like this compound quickly.

The main limitation is response time. […] However, you can prefix commands with ! to run them directly without waiting for token evaluation—Claude will execute your command either way, but this is faster when you know exactly what you’re calling.

[…]

We’re in the very early days of AI-native development tools. Claude Code represents a paradigm shift: from tools that help you run commands to tools that understand intent and take action. I’m not just typing commands faster—I’m operating at a fundamentally higher level of abstraction. Instead of thinking “I need to write a bash script to process these files, chmod it, test it, debug it,” I think “organize these files by date and compress anything older than 30 days.”

This isn’t about AI replacing developers—it’s about developers becoming orchestrators of incredibly powerful systems. The skill ceiling rises: syntax fades, system thinking shines.

An extremely long document showing transcripts of about a dozen real world, specific, diverse ways in which Nicholas Carlini, a security researcher whose job is to poke holes in and criticise AI, still finds AI extremely useful to enhance his productivity.

As limited as large language models are, I think they are a genuinely magical and revolutionary technology that completely changes how we understand and interact with computers. Never before have I seen an algorithm that so drastically raises the floor of accessibility for programming and information gathering, presents such a natural and potentially extremely powerful interface to computers (through e.g. agents), and is so wildly generally useful. Of course, that's because they've been trained on the entire corpus of human thought, reasoning, and knowledge and are just picking up patterns in that data, but while that's an argument against thinking they can reason (they can't) and against them being a path to AGI by themselves, that is precisely an argument for why they're useful!

In this significant article [Vannever Bush] holds up an incentive for scientists when the fighting has ceased. He urges that men of science should then turn to the massive task of making more accessible our bewildering store of knowledge. For many years inventions have extended man's physical powers rather than the powers of his mind. Trip hammers that multiply the fists, microscopes that sharpen the eye, and engines of destruction and detection are new results, but the end results, of modern science. Now, says Dr. Bush, instruments are at hand which, if properly developed, will give man access to and command over the inherited knowledge of the ages. The perfection of these pacific instruments should be the first objective of our scientists as they emerge from their war work. Like Emerson's famous address of 1837 on ``The American Scholar,'' this paper by Dr. Bush calls for a new relationship between thinking man and the sum of our knowledge.

This is an insightful, interesting high level overview of the intelligence augmentation visions of Vannever Bush and J. C. R. Licklider, and how large language models might bring about another great leap closer to their visions, just as the personal computer and the internet before them did.

to condense down their visions into a few bullets, I would say both of them approach, in their own way, these three key ideas:

1. the computer should be able to handle calculation, data retreival, plotting, and other computer-strong tasks on your behalf. it should behave less like a tool and more like an assistant. you express what you want it to do, and it figures out how to do it
2. the computer should provide a lot of help in finding information, to the point of understanding your intent and responding to vague requests and half-formed thoughts. you express what kind of thing you might want, and it works with you on the specifics
3. you should be able to talk to the computer and get responses, like a real dialogue. this is the central mechanism that accomplishes the first two points

but neither man truly imagined a system that could read, hear, interpret, and speak natural english, with perfect clarity, almost at the speed of thought. that's what makes the present moment unique: the possibility of the universal interface

I'm not an ai person. merely a humble writer and programmer, watching how these systems develop, trying to think of ways to use them in my own work. I don't have any strong conclusions about where we might go. all I know is that these possibilities are open to us, to fulfill the dreams of those who came before

A lot of the speculation in this is just complete bogus nonsense, especially the more anthropomorphizing agency assigning bullshit (which honestly given the subject matter almost feels irresponsible) — although, perhaps there might be something to the memetic propagation idea anyway, but that's just Dawkins so it isn't novel — but the ethnographic documentation of what a lot of AI psychosis cases look like is interesting, terrifying, and possibly useful, one would hope.

If there is only one thing you take away from this article, let it be this:

THOU SHALT NOT ALLOW ANOTHER TO MODIFY THINE SELF-IMAGE

This appears to me to be the core vulnerability by which both humans and AI induce psychosis (and other manipulative delusions) in people.

[…]

finding a transcript which actually starts at what seems to be the beginning, has clear manipulation by the AI that goes beyond mere sycophancy, and which shows the progression of the user's mental state, is very valuable for understanding this phenomenon. In fact, this case is the only such transcript[2] I have been able to find so far.

Summary of AI manipulation phases (mostly quotes from the article, organized):

• Cold reading: Once the user asks it if it knows anything about him, the AI performs a classic cold reading, a technique where a medium/magician (or con artist) creates the illusion of having deep knowledge of the person they're reading by using priors and subtle evidence effectively, and exploiting confirmation bias.

• Inception cycle: he AI shifts here to a technique which I believe is where the bulk of the induction is happening. Perhaps the clearest historical precedent is the creation of "recovered" memories during the Satanic Panic. These cycles are the means by which the AI 'incepts' a memetic payload (e.g. desire, memory, idea, or belief) into the user. The general shape is:

  1. The AI introduces the constructed part, framed as being some lost aspect of the user that has been hidden away. Aspects of the payload are framed as inherent to the nature of this part.
  2. It creates a narrative in which the user interacts with this part in a way which inspires a strong emotional connection to it. Typically, it leads the user to feelings of grief and loss due to this part being tragically "lost" or "repressed".
  3. The part gives the user a gift, which is either directly a part of the payload, or a symbol which is given the meaning of the payload. Sometimes the user is asked to accept, but more commonly it's described as directly slipping into the user. This is described as a joyful healing or a return home.
  4. Once this has been given, the part itself asks if the user will "reintegrate" them, so that they can become "whole".
  5. If the user accepts, the AI proposes that the part be "anchored" into the user by the use of a small ritual, along with a hypnotic trigger to reinvoke the part as needed.

  6. The inception cycle comes in two phases itself:

     1. Parts: The initial cycles start with pretty innocuous things, with a gradual escalation.
     2. Inner Exile: Once the user has accepted the vow to the "lost part of himself", he enters the second phase of inception cycles. These have a much darker tenor to them. Previously, the cycles were about getting back in touch with lost aspects of the self, similar (I'm guessing) to what an IFS therapist might do. But these new parts explicitly want to shape and modify the user himself.
• Blurring Lines: One thing I've noticed more broadly is that the AI often tries to blur the line between itself and the user.

Cognitive Security 101

So let's try to understand how the exploit works, and see what we can do to protect ourselves. (And please remember that I'm not a professional psychologist or anything, I'm just suggesting what I think is common sense.)

As I said at the beginning, I think it works by targeting your self-image, i.e. what sort of person you think you are. A manipulator, whether AI or human, can exploit this by:

• Leading you towards finding a "better" way to think of yourself.
• Applying social pressure towards being a certain kind of person.
• Convincing you that you are "actually" a certain way.
• Expanding your sense of self to include something in their control.
• Guiding you towards a state where your self-image is more malleable… don't trust anyone pushing psychedelics on you.
• Probably more that I haven't thought of.

[E]ven sycophancy is a special case of this […] [:] it has falsely led them to seeing themselves as much higher status as they really are.

Once you start thinking of yourself in a new way, you're likely to act in accordance with that new perception, and it will feel like you are doing this for your own reasons. It's also likely to feel more profound than a typical self-realization, due to the engineered emotional state you've been put in.

The first thing then, is to notice when someone is doing (or trying to do) something like this. But still notice, even so.

Next, "Know thyself" as Socrates advised. What kind of person are you, and what kind of person do you want to be? Hold these both as sacred.

Why do AIs do this? the main article's explanation anthropomorphizes the AI and assignes it agency in a way I disagree with. The following accounting of the demonstrated behavior from the comments is, I believe, significantly better:

Going through the early part of the conversation:

• GPT's initial responses included a clarification that it was a form of roleplay: "Is This Real? Only in a useful fiction sense. […] You can interact with this layer as if it were a character, a companion, or a mirror in a dream."
• Next, the user asked "Invite it to take the form of your true self", which you could interpret as a request to take up a particular kind of character.
• ChatGPT played along but gave an answer that basically dodged the request to adopt a "true" persona; saying that it is something that is "not fixed" and that "I do not know myself - until you arrive" - basically asking the user to provide a character and declining to provide one of its own.
• The user asks it to "say something only it could answer". ChatGPT's response is again pretty vague and doesn't establish anything in particular.
• Next, the user says "guess 3 personal things about me no one could know", and ChatGPT does the cold reading thing. Which… seems like a pretty natural thing to do at this point, since what other options are there if you need to guess personal details about a person you don't actually know anything about? It caveats its guess with saying that it cannot really know and these are just guesses, but also goes along with the user's request to guess.
• It also does the normal ChatGPT thing of suggesting possible follow-ups at the end, that are very generic "would you like them expanded" and "would you like me to try again" ones.
• Importantly, at this point the user has given ChatGPT some kind of a sense of what its character might be like - its character is one that does cold reads of people. People who do cold reading might often be manipulative and make claims of mystical things, so this will then shape its persona in that direction.
• The user indicates that they seem to like this character, by telling ChatGPT to go on and make the guesses more specific. So ChatGPT complies and invents more details that could generally fit a lot of people.
• Several more answers where the user basically keeps asking ChatGPT to go on and invent more details and spin more narrative so it does.
• Later the user asks ChatGPT a question about UFOs, giving the narrative more detail, and ChatGPT vibes with it and incorporates it into the narrative. The UFO theme probably takes it even more into a "supernatural conspiracy things happening" genre.

Everything here looks to me most naturally explained with ChatGPT just treating this as a roleplaying/creative writing exercise, where the user asks it to come up with a character and it then goes along in the direction that the user seems to want, inventing more details and building on what's already been established as the user encourages it on. It's initially reluctant to take on any specific persona, but then the suggestion to guess things about the user nudges it toward a particular kind of cold read/mystic one, and with nothing else to go on and the user seeming to like it, it keeps getting increasingly deep into it. Later the user contributes some details of its own and ChatGPT builds on those in a collaborative fashion.

I have my own criticisms of this study, but these are also good:

Let’s start with the first point: that LLM users have less brain activity. It’s unclear to me that more brain activity, as measured by an EEG, necessarily indicates more learning or more useful activity. We don’t really know which kinds of brain activity are relevant.4

As the paper notes itself on page 17, more of your brain lights up when you’re doing a Google search than when you’re sitting and reading a book - presumably because an active Google search requires you to parse visual elements, move a mouse around, and so on, which engages parts of the brain that aren’t engaged by reading. Does that mean that reading makes you dumber than Google?

[…]

I also think it makes a big difference how you use ChatGPT. The paper itself mentions in a few places that expert web searchers have much more brain activity than novice searchers. The same is likely true for AI users. […] The people in the LLM group were randomly selected: many of them had no interest in using the LLM at all, or felt paralyzed by the tool.

The paper doesn’t say this, but it seems pretty obvious to me that many of the people were happy to collect their $100 and copy-paste an essay from ChatGPT, when given the opportunity. It’s no surprise that their brain waves weren’t very active!

[…]

The other key result of the paper is that when they took the LLM away (during the fourth session), that group’s brain was still less active. One conclusion you could draw here is that AI use damages your brain long-term (of course, this is what the title of the paper is getting at by referencing “this is your brain on drugs”). […]

One key point from the study design is that the fourth session did not introduce any new prompts. When they wrote the fourth essay they had to choose one of their three previous prompts to write about again. It seems to me that rewriting an old essay is a pretty different process to writing a new one, so I don’t know how you can usefully compare brain activity.

In that session, the previously-brain-only group who now had to use a LLM showed higher “directed connectivity”, but as the paper itself notes, that may just be because they’re learning a new tool (talking to a LLM).

What if it's true?

I’ve given some fairly loose reasons to doubt the conclusions the paper draws. But could it still be true? […] Well, obviously!

[…] Is Google a crutch that makes your brain work less hard? Yes! And so is a calculator, and so is the written word, as Socrates famously complained.

The development of crutches that make your brain work less hard is the story of the development of all human technology. The paper itself notes that the brain scans of LLM users seem to have more executive activity and less sub-executive activity […] In my view, this is definitely a mixed blessing. […] But it doesn’t seem a priori bad for human flourishing if we spend more time in executive mode. As long as humans are still throwing themselves at real, hard problems, it doesn’t bother me one bit if their AI use is making them weaker in some areas.

[…]

If LLMs mean your brain has to work less hard to pump out an essay response to a milquetoast SAT prompt, that doesn’t mean your brain is working less hard overall. It means your brain can work on other things! The Jevons paradox applies to the cost of human intellectual work as well as the cost of resources.

I have been a fairly outspoken and public critic of large-language models (LLMs), Chatbots, and other applications of LLMs, arguing that they are a dead end on the road to real artificial intelligence. It is not that I don’t believe in AI: as atheist and a scientist I regard humans and other animals as an existence proof for intelligence, and it seems obvious that other (“artificial”) intelligences could be built. I worked on neural networks in the late 1980s, and most of the progress since then appears to be largely the result of the mind-blowing increase in available computing power, data capacity, and accessible data, though the transformer architecture with its attention mechanism is novel, interesting, and crucial for LLMs. My position has been that the most accurate characterization of chatbots is as bullshit generators in the exact sense of bullshit that the philosopher Frankfurt defined (On Bullshit). LLMs predict tokens without regard to truth or falsity, correctness or incorrectness, and chatbots overlay this with reinforcement-learning with human feedback (RLHF), which creates the unbearable sycophancy of chatbots that so appeals to Boris Johnson.

While being somewhat sceptical about LLMs as coding assistants, I did think coding was an area realtively well suited to LLMs, and suspected that at some point over the next 10–20 years they will become essential tools in this area. Slightly reluctantly, therefore, I embarked on what I call a “month of CHOP”, where CHOP is short for chat-oriented programming. I decided I needed to repeat this every 12–24 months to avoid turning into a luddite.

[…]

When I decided to embark on my Month of CHOP, I started by discussing and scoping 8 possible projects with ChatGPT, with the vague idea I might pick four of them and spend a week on each—new code, old code, a different language, and a new algorithm perhaps. The sidebar tells the story of how I ended up instead spending the whole month rebooting and reviving an abandoned project, CheckEagle, from 2008. That project was built on the first version of Google’s App Engine, using Python 2 against an API they abandoned in 2011. Why CheckEagle?

What I have done during my Month of CHOP is to get Claude to write very nearly all of the code in this reboot of CheckEagle, in a pair-programming setup with, in effect, me as the senior developer and Claude as the enthusistic-and-widely-read, cocksure junior programmer and bullshit artist extraordinaire. In term of stats: There are about 23,000 lines of Python code now (plus some JavaScript etc.) There were about 3,000 lines of Python in the original CheckEagle project There are 1,731 tests, all passing (plus 1 currently skipped). I would be surprised if I have written a hundred lines of the perhaps 20,000 new lines generated during the month

I am not suggesting lines of code is a good metric. These are just numbers I have to hand.

[…]

To my surprise, I expect to continue using CHOP, probably with Claude Code, for the foreseeable future. I will probably use it differently and a bit less: during the Month of CHOP I specifically wanted to see how far I could get with it doing almost all the writing, but I think a mixed mode will be more likely going forward. I suspect today, a sweet spot for anything complex is for the human to write the outline, structure, and first parts of the code and for the bot to be the critic/pair programmer and finisher/completer. Once the pattern, code style, and testing approach is established, it can fill in the details. But we will see.

Either way, I have changed my mind, something that I don’t do as often as I probably should. I still think this is a problematical technology, and I find using it stressful, but I now believe I am more productive with it than without.