An inconvenient truth (for Lispers)

The claim that Lisp is an isolated environment that cannot interact with the modern world or be easily deployed is technically incorrect.

Common Lisp’s Foreign Function Interface (FFI) is often superior to modern mainstream languages; for instance, SBCL’s FFI is significantly more efficient and less cumbersome than Go’s cgo, which requires heavy context switching and stack management in comparison to essentially a single CALL instruction.

Furthermore, the deployment "issue" is a misunderstanding of image-based development. Using ASDF to create documented, repeatable builds from scratch, that are self sufficient and thus work identically on the developer's machine and the prod server, provides a level of reproducibility and environmental consistency that essentially functions as a more efficient Docker. And if you need debuggability and observability of the distributed running image, well, then, Lisp has Docker's "start a bash shell in the image and hope it has the commands you need" approach by a mile.

If a language can talk to C better than Go can and can ship repeatably and in a self contained manner better than Docker, it is not an island.

The argument suggests that Lisp fails because it makes it so easy to write an 80% solution that fits all their needs that nobody feels the need to standardized around one or a few polished options and work together to complete them, resulting in an ecosystem of unfinished 80% solutions instead of one polished 100% solution.

This falls apart upon logical inspection: if Lisp makes building the 80% of a library that you need in your specific case so trivial, then the lack of a 100% library is not a problem: an ambitious developer could simply spend the small amount of time required to build the specific 80% that has everything they need for their project and proceed. If the "tax" to create a library is that low, a lack of "standard" libraries cannot be a legitimate blocker for building massive, impactful projects.

It's also just empirically false. As discussed in this essay by Hayley Patton, the Common Lisp community does tend to centralize on one standard, 100% implemented — or as near as their limited resources can get — solution to most problems, including things like static types, pattern matching, build systems package managers, and so on; and before they do, the added ability to quickly explore various lines of flight in the problem space, in a decentralized, deterritorialized manner, then form a common understanding through highly interconnected, rhizomatic communication over a network about what the best approaches are, instead of prematurely standardizing on one implementation, is actually an advantage. So it's unclear that this explanation for the Lisp Curse is really a problem.

Critics argue that Lisp projects are impossible to maintain because they become "private languages" full of custom macros and DSLs.

However, every large-scale codebase in any language—be it C++, Java, or Go—is effectively a private language. Every project develops its own internal model of the world, its own terminology, and its own unique abstractions that new hires must learn. A well-designed Lisp DSL is no more difficult to grasp than the complex object hierarchies or design patterns found in a massive enterprise codebase. The "private language" is a feature of scale, not a unique flaw of Lisp. And if the argument is that Lisp gives you more leeway to shoot yourself in the foot making incomprehensible abstractions, I'll just point you to EnterpriseFizzBuzz and C++, and yet infinitely more gets done with Java and C++ than Lisp.

This argument posits that Lisp programmers are too individualistic to contribute to large, collective projects (either for personality reasons, or technical reasons within the language (see the 80% section and the private language sections above)), leading to a fragmented ecosystem.

The reality of the software industry contradicts the idea that this "soloist" nature would prevent the creation of impressive things. There are numerous "one-man army" developers who have built world-changing, high-impact projects (like Redis, SQLite, or FFmpeg), so why don't we see any world-changing, high-impact soloist projects using Lisp? If the problem with Lisp is only that you can't get big teams to work together using it, then that shouldn't prevent such soloist projects from arising, and yet they don't, so it must be something else.

Moreover, if Lisp were truly a "God-mode" force multiplier for the individual, we wouldn't just see some high-impact soloist projects using Lisp: we would see most of them being Lisp, because the Lisp ones would make progress quicker, and thus gather usage and contributions and attention faster, and out-compete everything else. Instead, even for solo work, we see "boring" languages winning, and nothing very meaningful using Lisp.

The "Beating the Averages" essay popularized the idea that Lisp is a "secret weapon" that gives companies a massive, invisible advantage.

This is an unfalsifiable cope. It's basically saying "trust me, companies that are successful with Lisp totally exist, they're just invisible."

Worse than that, there are a lot of reasons to think that this isn't what would happen if Lisp were as productive as we think:

1. Companies tend to advertise what gives them a competitive advantage, not keep it a secret, when it's not a trade secret (like Coke's recipe) but something that's already out there and accessible and known about, because the cat's already out of the bag and it's a great way to advertise to investors, stockholders, prospective employees, and even customers!
2. The Lisp community carefully documents and treasures every Lisp success story — Viaweb, ITA Software, Grammarly, AutoCAD —, whereas if Lisp was kept secret because it was such a powerful weapon, then we wouldn't know about those, now would we? You could argue that it's a time lag to the reveal of Lisp-based companies, but I don't think ITA or Grammarly or AutoCAD were hiding it, and even if we assume the whole rest of the industry is following the Viaweb model, then, well, we've had like 30 years for more retrospectives to come out, and we haven't. And we actually do know what technologies the most successful projects and companies are using, because they're hiring and/or building in the open.
3. This doesn't explain where all the open source major projects with Lisp would be. Sure maybe companies would keep their "secret weapon" (a 50-year-old programming language every enthusiastic hacker and Paul Graham-reading startup founder knows about) secret, but open source projects would want to choose whatever tool is most effective for the job, and obviously build in the open.

The last, final argument that long term Lisp cultists will fall back to, when all others have failed, essentially amounts to a conspiracy theory: that Lisp makes programmers nonfungible — because they can use the language in such advanced ways to get things done, due to its high skill ceiling, that if you need someone else to work on their project, you need another person with a very high skill level to replace them, and even then, it'd be hard to onboard anyone — and also gives programmers much more leverage — in terms of how productive they can be, and what they can do alone — and this is a threat to corporate managers, who want to suppress programmers as a labor force, and turn us from independent artisans to assembly-line craftspeople.

I've talked before about why I find this argument offputting and unsatisfying from a craftsperson perspective; to sum it up, I think that we shouldn't want our irreplaceability to come from writing incomprehensible towers of abstraction so that nobody else (probably including us later down the line) can understand what we're doing, but from clear, real, specific wisdom, expertise, experience, and skill in the domain and tacit knowledge about the project underway, and I think that in actuality — and even with Lisp — this is generally true anyway, so I don't think Lisp is that much of a threat managers that they explicitly choose against it.

But I think there are other reasons this idea is wrong:

First of all, I think that a simple, standardized language with only one way to do any given thing can also give programmers higher leverage! It does this by reducing cognitive load deciding which features to use, or designing a custom macro DSL for the problem, which can instead be focused ruthlessly only on solving the problem itself, by reducing complexity budget used up by abstractions and DSLs that might turn out to be wrong, and by making it so that any programmer has leverage anywhere: every programmer becomes infinitely more useful at the same time that any programmer becomes more replaceable.

Secondly, there's the question of FOSS projects. There are plenty of FOSS projects, big and high impact ones, that aren't dependent on, or controlled by, corporate interests, such that they'd be forced to use "corporate" technologies. In fact, the whole free software movement is fundamentally opposed to corporate influence in a lot of ways. And the argument that most open source programmers would just blindly use whatever language or technology they use in their day job on their hobby projects and FOSS contributions just because they have no imagination is extremely condescending and belittling — many FOSS contributors, especially those that start projects, are language polyglots with strong opinions; and a majority of open source developers readily voice that they contribute to FOSS and use technologies different from the mainstream corporate world specificalily to escape the daily drudgery of the technologies they know from work; often, they actually contribute to or start FOSS projects specifically to learn new technologies, and spend a lot of their free time learning such new technologies. Do you think developers, if they really found using non-Lisp languages, such grueling hard work, wouldn't jump at the chance to use something else in their free time? So why do no such projects that use Lisp get much of anywhere and become really successful?

Even if most FOSS contributers are such dull, unimaginative apes as Loper OS and his ilk believe them to be, wouldn't small teams of Lisp programmers — surely there are enough of them — so clearly out-compete, due to the supposed productive advantages of Lisp, the herds of boring, drudge-work FOSS programmers, that their projects would quickly become visible, influential, and useful? But when's the last time any of us actually used FOSS Lisp-written applications or infrastructure? Sure, Shinmera wrote a whole video game in CL by herself; but guess what? That happens all the time, even from SDL or engine up, not using Lisp.

I also think this is wrong because it gets the causality backwards, or at least pushes the question back a step: managers don't choose Lisp because there aren't a lot of Lisp programmers, and so they won't be able to hire more, or replace the ones that leave or "go down without being rebootable", yes; but if there were plenty of Lisp programmers, I don't think managers would have an issue with using Lisp and hiring Lisp programmers. And if there were plenty of high-impact open source projects (collective or soloist with contributions) and incredibly effective startups using Lisp, or even if it just made programmers much more productive individually for their hobby projects, then we would expect to see plenty of Lisp programmers in the wild looking for Lisp jobs, and building open source Lisp infrastructure companies would want to hire people to work on, which would create the basis for healthy Lisp use in the industry. This is exactly the pattern you see e.g. with Rust; a lot of people are using it for their, often high impact, hobby projects, and you can see how it's now migrating into corporations. So the question returns to, where's all the Lisp usage?

This basically laves three possible conclusions:

1. Lisp remains "Potential Man" because of path-dependency
2. Lisp isn't as much more productive as we think

I personally think the most likely solution is probably a combination of both.

Regarding the last option, I think that for a language to be significantly more productive and capable than its competitors, it can't just rely on being nonspecifically "more powerful" with general purpose tools like macros or image based development; it needs to fill a specific niche with a specific set of features nothing else can match. Rust has borrow checking and an ML family type system that represents memory and performance decisions at the type level for maximum performance and safety; Go has an incredible concurrency model and completely standalone static binaries; SQL elegantly expresses relational algebra; etc. This is because, while something like macros or image based development might make you 2X more productive overall, a unique capability can make you infinitely more productive, by making things that weren't feasible before (either impossible, or just not worth the effort, or too hacky) suddenly possible. Even if it doesn't make you infinitely more productive, though, a unique feature can, because it's much more focused and specific, still have a much greater productivity multiplier than features that just vaguely, nonspecifically "are more powerful."

Now of course, some general purpose dynamic languages reached extreme popularity, like Python. Why didn't Lisp? I think that's partly where the path dependency comes in: by the time the world was ready for a more slow/heavy dynamic language, Lisp or Scheme was already extremely old and crufty, with a bitter and lonely community, so a new language came along to fill that void.

I also think that the hardest part of programming is understanding the problem domain: cutting it at the joints to create a powerful ontology and description of the behaviors of things, then designing the right goals in light of that, then planning your architecture, algorithms, and data flow to best achieve those goals. All of that happens at the conceptual level, at the level of intuition-building, not at the code level, and as I argue elsewhere, I think macros extremely rarely offer an advantage in translating that domain models to code over the features most modern languages already have to speak about domain models as a first class language of objects and verbs — such as structs, methods, functions, higher order functions, generics, decorators, ADTs, and so on — especially since macros just translate to the same ontology underneath anyway. They just help you save a few lines of code, and I don't really think that's that important for the model translation aspect, which I think itself is less important that the building of the understanding and the model in the first place, which happens mostly in the programmer's head.

The second hardest part of programming is the blue-collar drudgery: the compliance, documentation, polishing, handling carefully of edge cases, usage scenarios, and combinations of tools. This, again, isn't something Lisp's unique features make easier.

I still think that Lisp has some incredible features that other languages just can't really beat. The problem is that those features have been whittled down over time:

1. GC is now mainstream
2. so is hot reloading
3. so are higher order functions
4. so is recursion
5. so is reflection
6. so is standalone images via static binaries
7. so are macros, or other means of metaprogramming that get 80% of the way there (via e.g. passing blocks to functions, operator overloading, comptime, decorators even)

What remains are advantages that are much more limited and contextual: macros may be extremely helpful sometimes, in a way you can't get from anything else, but that's a very limited set of cases, compared to the vast majority of the tasks you'll perform as a programmer, for instance.

I also think I'm just coming to realize that a lot of other languages now reciprocally have unique, incredible features Lisp can't beat, and that the remaining features Lisp has just aren't as important to real world productivity as those unique features of other languages, because of the generalist vs niche problem.

This doesn't change my appreciation for Lisp. This is more of an attempt to understand, as part of a series of other posts, why I just don't find it that productive to use, compared to other languages, and have generally been unimpressed with it in practice. Other posts:

• Language doesn't matter. Libraries do.
• Reconsidering Go
• Do we really need macros much?
• The only thing Lisp fails at