UNIX, Lisp Machines, Emacs, and the Four User Freedoms

• The freedom to run the program as you wish, for any purpose (freedom 0).
• The freedom to study how the program works, and change it so it does your computing as you wish (freedom 1). Access to the source code is a precondition for this.
• The freedom to redistribute copies so you can help others (freedom 2).

• The freedom to distribute copies of your modified versions to others (freedom 3). By doing this you can give the whole community a chance to benefit from your changes. Access to the source code is a precondition for this.

  — What is Free Software? — GNU

Any true hacker knows these principles by heart. Even though I hold many disagreements with Stallman and the FSF-hardliners — for instance, I see free vs proprietary software, and the software freedoms, as a matter of structural critique and industry ethics, not personal purity; proprietary software is being unethical against me for the way it treats me, I am not unethical for using it, because I am the one on whose side the rights lie — I agree with them, fundamentally, on so many more issues than I disagree with them on. We share the same ultimate values and goals, even if sometimes our methods diverge, even if not all of us can be pure enough to only use the MNT Reform and the Librem 5, or live up to Stallman's hard-line fervor.

So it is undeniably gratifying, not to mention dizzying, to look back at the massive progress the free software movement has made since that first announcement on the unix-wizards USENET group in 1983, long before I was born. What a fight it has been, and how far things have come since then! GNU/Linux is basically viable for anyone at this point, for one thing — and that matters a lot, even if the Year of the Linux Desktop never comes, because it thus offers a refuge to those like us who value choice and software that servers our own ends, instead of the ends of our corporate (or state) masters. Even if much of that progress has come in the guise of free software's more limp-wristed corporate cousin, "open source," in the end, it still means that more of the software people actually use than ever supports these freedoms more than it ever has before, and more viable free alternatives exist to proprietary software.

There's just one thing that nags at me, one thing that casts a pall over all the wonderful progress we've made.

I think the free software movement's whole fundamental software stack betrays the spirit of these freedoms, and until we fix that fundamental betrayal, the meaning of the four freedoms won't be clear, and their import for the average user will be a joke.

Why?

Because UNIX won.

I used to love UNIX. I used to think the "UNIX philosophy" was one of the last words in operating system design, a shining pillar of right-thinking software engineering the only flaw in which was that it wasn't UNIX enough — as many Plan 9 fans seem to think — and that anyone who questioned the tenets of its philosophy was likely an architecture-astronaut moron. That Windows was bad simply for not being UNIX, and that Linux was likewise flawed simply because it wasn't sufficiently UNIX in some way or another. I used to look on the BSDs in jealousy, and fantasize about using them if only I didn't need various minor things like applications and modern hardware support.

Then I got more experience with UNIX, and a glimpse of the alternatives, and began to hate it.

More than that, I began to see how it was what made my ideals laughable for the average user (including me).

UNIX's flaws run deep, and they were known even at the time, so how did we end up in this situation in the first place? When Stallman was first brainstorming what would later become the GNU Project, he initially considered basing his nascent operating system on the Lisp Machines he'd had at the MIT AI Lab, but eventually dismissed the idea:

At first, I thought of making a Lisp-based system, but I realized that wouldn't be a good idea technically. To have something like the Lisp machine system, you needed special purpose microcode. That's what made it possible to run programs as fast as other computers would run their programs … so I rejected the idea of making a system like the Lisp machine.

I decided instead to make a Unix-like operating system that would have Lisp implementations to run as user programs. The kernel wouldn't be written in Lisp, but we'd have Lisp. So the development of that operating system, the GNU operating system, is what led me to write the GNU Emacs.

— My Lisp Experiences and the Development of GNU Emacs — Richard Stallman

In fact, according to James Gosling and many others, up until the GNU project began, Stallman vociferiously disliked UNIX. But, in GNU's case as with many a contemporary, UNIX won out — not because it was more powerful, or more stable, or easier to use, or more reliable, or better-designed, but simply because it was there, it was cheap, and because at the dawn of the mini- and then micro-computer age the hardware at the time put inviolable limits on what could be made to work in the first place, limits that simply ruled out most of the other alternatives — and of those that were left, UNIX happened to be the least-worst.

Do not confuse the vagaries of historical context and accident for the inevitable hand of fate determining luminous truths to be upheld for all time, however. That UNIX was perhaps the only viable option at a certain point in history due to hardware limitations says nothing about its general merits outside that context, or the applicability of its design principles to modern software. In actual fact, UNIX has a great many flaws, most of which are not accidental, but inherent, stemming from its religious dogmas themselves. Like all religious texts, The Art of Unix Programming seems to contain some good ideas — even ones that are quite contradictory with each other, or with the how the religion seems to have been put into practice, or with the messages its adherents seem to take from it — but it is at the same time also deeply flawed, especially in what its dominant adherents seem to glean from it, which seems to be summed up by Doug McIlroy's quote:

This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.

Point by point:

• Text is the universal interface? Hah, don't make me laugh. Text has no structure, no semantics, no meaning or metadata, but program data and communication does, so the most it can hope to be is merely a medium for programs' actual interfaces: if you leave text as the sole interface protocol, every program is forced to construct from first principles, Crusoe-like, its own method of expressing its structural, semantic data on the desolate beachhead of plain text streams, and that encoding becomes its actual interface. As a result, nothing will be able to communicate without ad hoc munging — with which the system doing the munging can't even help you, since all it sees is a raw text stream — with tools like grep, sed, awk, and cut. The belief that simply because things communicate with text, that any kind of common interface is to be had, instead of merely a common transmission mechanism, is laughable. It's like saying that simply because everyone communicates over radio waves, that everyone has a common jargon or language.

  Now, is having a common transport medium good? Is it good, even, that the transport medium is as simple, universal, and (most importantly) human-readable, as text? Perhaps, if we take the world UNIX has built for us as a given, but that's simply begging the question. There is admittedly a tradeoff here — the more rigid and specific you make your system's common interface, the less is up to each program to define in their ad-hoc interfaces, and the more interoperable things become, but the harder to encode and decode that format becomes, and the less likely things are to adhere to it, and the less flexible it is, whereas the more general and flexible you make your common interface, the more interface definition is pushed into the hands of each individual program, and the more entropy gets ahold of everything. But from first principles, in my opinion, the best way to balance this tradeoff would be to select or define a data structure that is simple and flexible enough to be easy to encode/decode and represent any sort of data in, yet still structured and imbued with semantic information enough that the general facilities for manipulating it built into your system can be reasonably high level and advanced, and more basics of how things work are held in common, and then to simply write the system that surrounds that interface such that it can render it reliably and without information-loss as something human readable, like a table or a graph. Something like JSON, or Lisp's plists, or the tables of nushell might do the trick. Perhaps even a rich library of different data structures could be used to communicate, like in oil shell. As it stands, however, UNIX fell completely off one end of the balance beam, and made its common interface so universal, so general, that it wraps back around and becomes essentially not that universal at all.

• Write programs to work together? This is a good principle, in theory, but in reality UNIX hobbles this capability by treating text streams (or worse yet, byte streams) as the "universal interface", meaning that without copious esoteric arguments (like -print0 on find and the associated -0 for xargs) and merciless string-munging, most programs do not, actually, work together very well, and it is difficult to make them do so, for lack of high level tools for manipulating their input and output, since all the command language (shell) has a concept of is raw text streams, nothing higher level than that. The closest to something good in this directly is awk, but it is very telling that actually properly getting programs under UNIX to talk to each other requires its own relational-database-esque tabular-data-processing sub-programming language, when passing data between functions in a pipeline in any proper language, like OCaml or Lisp, requires using nothing but the powerful and high level data manipulation abstractions built into the language, no character-by-character sewage-wading necessary.

  Moreover, why design separate single-purpose programs to work together, when you could have your shell language and your OS and application implementation language be the same, and present everything, not as little programs with awkward and idiosyncratic interfaces, but as libraries that the powerful high level language you use to control the system can command at-will? The integration would be inevitably far better.

• Do one thing and do it well? To do one thing truly well, you have to solve the whole problem from first principles, pull it up by the roots, which often leads to programs that are larger and more far reaching than any UNIX advocate would be willing to countenance. Indeed, oftentimes in the process of trying to solve one problem well, you have to also solve many peripheral, deeply intertwined and interrelated problems, in order to give ergonomics and completeness that won't make your users want to die. Give me programs that each solve an entire problem domain, a complete task, with aplomb, and predict my every need in each area! ""

  You don't even have to give up composability, programmability, or inspectability to have that, either: just design your programs, as I'll cover, to expose the internal state, data structures, functions, classes, etc, that they are composed of for live, run-time programming and modification by the user, and any program can easily become scriptable, no matter how monolithic. What about modularity? If every program is trivially modifiable, extensible, easy to mold to your whims and needs, if every program is spinning clay in your hands, modularity is easy, without the need to break everything up into tiny pieces that communicate across low information-bandwidth boundaries and constantly fail and break at the borders, no matter how reliable each individual piece is: you can achieve modularity within the code itself. But the fatal flaw of the technology UNIX was built on — as we will see in a moment — is that it makes programs inherently difficult to script and control, inherently monolithic, so that breaking what should be a deeply integrated system that communicates constantly and intelligently with itself into a thousand tiny programs becomes necessary.

Perhaps, despite these flaws, we could at least say that:

1. UNIX was the best option in its particular context, even though that context has long since passed,
2. UNIX did have the inklings of some excellent, very good ideas, namely the idea of a thin-waist common interface between programs that is roughly human-readable, making programs programmable and scriptable (even if they went about this the wrong way), and that simplicity is at least a value to consider.
3. In today's landscape, UNIX is somewhat "the worst operating system, except for all the others." NT had a few neat ideas with COM and PowerShell, but it seriously isn't great as far as I can tell (I know little about it, truth be told).

The problem is that the context that made UNIX truly a good option is long gone, and all that has been left behind is a cult of cave-trolls obsessed not with the functionality or usability we get for the size or processor-intensiveness of our programs, but with small size and low CPU utilization as goals in themselves, gods on whose alters we should and must sacrifice actual usability, reliability, handling of edge cases, and useful features — that, and an industry that refuses to take what was good about UNIX, and innovate on that. UNIX is only the best operating system excepting all the others because the better ones were ahead of their time and died out, whereas UNIX was perfect for its time, and has now lived long past its life expectancy.

The greatest flaw of UNIX, in my view, however, is the technologies it was written for and still bears the marks of today — batch processing and the teletype terminal — and the technologies it was written with — C, primarily — and how they render the four software freedoms fundamentally farcical.

Those user freedoms — to inspect, modify, and redistribute the source code of the programs you run, don't really mean much on an OS like UNIX, because the impedance mismatch of actually making those changes and seeing their effects is astronomical. Because of the way our entire system is structured, and the straitjacketed languages we program it in, the process of software modification is laborous in the extreme. The hapless user must figure out what program any given behavior or command stems from and separately find and download the source code to whatever program they want to modify. Then they must manually search through that flat, dead text file codebase to find the source code relevant to the behavior they want, make modifications to it, blindly compile it, and then patch their system to use the brand new, if somewhat similar to the old, program they have just produced, instead of the old version. And if down the road someone upstream makes changes to that part of the codebase, good luck merging that. And you'll need even more luck dealing with updates, because now, for every program that you have modified on your system, you must maintain a separate fork: you must maintain your own changed copy of the codebase locally, pull in updates from upstream, make sure your changes still work, and recompile your program for yourself.

It is far from impossible, for sure, but it is far too much work to be done regularly, in a way that might meaningfully empower a user and make their lives easier. You don't do this sort of thing as a casual user that just wants to get shit done and simply wants to add a button for this or that command here or there — this is the sort of thing you get into if you either have absolutely zero clue of what is worth your time, and no life outside doing exactly this (if it is your main hobby) or if there is a big, dearly important change you need and you happen to have the copious time and skill on hand to do it. If you've ever considered modifying the behavior or capabilities of an open source program to better suit your needs in a way that wasn't already predicted and added as some hard-coded switch in a config file somewhere, you'll be intimately familiar with this logic — wondering whether it's really worth the time and effort to download that giant blob of C or C++ or Java code from wherever it is stored and poke around in it like someone lost in a maze. The ability to inspect and modify the code of your programs really doesn't mean that much in a situation like this.

Even interpreted languages like Perl or Python are hardly better: I still have to find the source code of the program, because it is not directly linked to the part of the program that's actually running that I can see or interact with, edit it as a flat, dead text file, instead of being able to easily see into its running state and its code as it is being executed, and then restart the program, and hope that nobody upstream changes anything. And very few significant applications or system components are written in Python.

In response to this dilemma, UNIX cultists of course have ready answers. They will proclaim that all of this is the result, not of the fundamentally static and dead batch-and-teletype-oriented technologies of UNIX, which force interfaces (both human and computer) to be text streams lacking any sort of real structure or metadata, which view compilation as the final irreversable entropic stage in producing a static binary blob that cannot be meaningfully changed or understood at runtime, which provide langauges only fit for the write-compile-segfault cycle, and which do not allow easy, accessible, and effective modification of a system at runtime… No, they will claim our problems stem simply from not UNIXing hard enough. Like a Marxist-Leninist that sees all the problems of failed authoritarian communist states as the result of simply not following the doctrines exactly enough, and fails to see how the doctrines themselves may have directly led to those failures to adhere to them, or that the doctrines themselves may have been fundamentally flawed, the UNIX cultists will refuse to acknowledge that there is any problem with the core ideas of the system:

The dumber ones will argue that if we simply sacrifice more usability, more reliability, more features, more completeness, more coverage of errors and edge cases, on the pyre of software anorexia, we can achieve a mythic enlightened state where our C software is so small and simplistic, and the tiny C programs that compose our systems are so disjointed and atomized, that patching a program and swapping it out for a different one will be trivial.

Some cleverer ones might suggest that distributing every program primarily as source code, which is then compiled in situ on the user's computer, is the solution to this problem: that way, at least, the source code is always easily available, and it's easy to replace a program on a system with a patched version.

Neither of these solutions solves the fundamental problems, however: the barrier to entry is still far too high, the feedback loop far too long, the system far too rigid. Even in the best-case scenario where programs are simple and source code is distributed next to them and every package on a user's system is derived from them — a world that would be massively impractical and inconvenient for most people who don't want to sacrifice every concievable useful feature and creature comfort of a modern system, and don't want to spend all day compiling programs and working through almost compiler errors with every update — the user who wants to modify anything still cannot directly change the program that they are actually running and see the effects immediately. They must still engage in the exhausting rigamarole mentioned above. And when software inevitably begins to be distributed as binaries, because it is simply more usable, and as software inevitably grows to deal with all the capabilities people need, and the creature comforts that people want to use, and to solve problems correctly and handle errors and edge cases and integrate better with other things, because it is fundamentally unreasonable to expect everyone to adhere to the UNIX cultists' ascetic hair-shirt approach to computing, things will only get more difficult.

But the UNIX cultists' very ideology of simplicity traps them with these inferior abstractions, because what they fail to see is that UNIX may be simple, but it is not simple in the way Scheme is simple, it is not simple in a way that truly empowers the people building on top of it, or the users directly interacting with it. It is simple the way a bat to the head is simple. UNIX provides paltry affordances, paltry abstractions — UNIX is a kludge that just happens to work well enough, and in enough environments, that it has spread like a virus because it is just barely "good enough" that nobody has bothered to innovate on it.

If the reason for this disconnect between the ideals of free software and how they actually work in practice isn't due to abandoning UNIX philosophy, but due to adhering to it, how did we end up in this situation? The real reason for this disconnect is that the four freedoms were not originally conceived with UNIX in mind at all. They are in fact incompatible with it, and find themselves fundamentally decontextualized by today's software landscape: Stallman's ideology was wholly shaped by the MIT AI Lab environment, an environment that no longer exists, and whose basic technological ideas and ethos have all but died out in the modern day, supplanted by the cuckoo's egg that is UNIX. It is from that alternate world of Lisp Machines (and Smalltalk machines) that his principles of software freedom stem. Only on them do they reach their full fruition.

On the Lisp Machines, code was transparently compiled in the background — compilation was an implementation detail, not an information-losing, entropic final publishing step — and deep information about the source code that corresponded to and produced every part of a program was always maintained, because of this lack of artificial separation between the program and its compiled form. The source code and the program that was running were fundamentally identical, or at least directly corresponding, in a way only interpreted languages today can even imitate a pale shadow of. Moreover, programs were distributed not as opaque binary blobs. They were distributed as compressed, perhaps slightly stripped down, images: dumps of the living, breathing, running system in RAM at the time the programmer decided the program was complete, not just dead source code and a separate binary, but snapshots in time, all unified, leaving nothing behind — source code, debug information, debug environment, running environment, were all one. This meant that the programs you ran, themselves came with the very source code that corresponded to them, and by modifying the source code from within that image, you directly modified the already-running program.

Even more importantly, Lisp was designed to let you dynamically shape and mold a program while it was running. Instead of modifying static, dead code, compiling, and then seeing the change, or experimenting with small fragments of code in a REPL to see what they might do in isolation, and re-running whatever fragments you had entered before each time you needed to redefine something, until you had a set of snippets you could finally expand into a complete program, which you would then debug much in the way you'd debug a compiled program, just with a little aid here and there in figuring out what isolated expressions might do from your REPL, the Lisp development cycle was entirely different. On a Lisp Machine — and still with Common Lisp and Emacs Lisp to a degree — you just hooked a REPL up to any running process, or began a running process, and were off to the races. You would shape and mold the program dynamically, as it was running, without the need to reload the program or lose state in any way — without even the hack known as "hot reloading" needed. You could redefine or add any function, class, variable, method, or anything else you wanted, and the program would just instantaneously shape to your will, you could see the changes immediately in the ongoing process, with no disconnect between what you were doing and your output or the program you were shaping itself. It was like doing shaping clay on a potter's wheel, or whittling wood, or some other hands-on, tactile, organic process. In comparison to this, even Python REPL is a mere pale imitation, linear batch processing shoved into a fancy dress, but not fundamentally changed.¹ And not only did this apply to constructing new programs, but it applied to modifying old programs as well, even programs that weren't yours. Just hook up a Lisp REPL to them and begin shaping and molding them to your will immediately. There was no distinction between OS, application, script, and shell implementation language to trip you up — just Lisp all the way down. Stack traces in the REPL would go all the way back down into their equivalent of the kernel!

And still more importantly, while in most programming languages it is very difficult to extend or change the behavior of a program without having to go in and edit the relevant parts of the source code directly — which is both hard, inconvenient, and means that modifications, especially from different sources, don't compose or stack well — Lisp offered many very powerful ways to modify the behavior of a function or class without needing to modify the original at all, ways that were, most importantly, modular and composable, based off lists of hooks or wrappers or extensions to functions or classes that could combine one with the other without necessarily crashing or producing an error, unlike trying to merge patched binaries or even patched source code where things have simply been directly modified. In essence, every single function and class and object in a Lisp program had its own extensibility API — the whole language was designed to make any and every program extensible by construction.

Already, you should be able to see how this makes the four fundamental software freedoms so much more meaningful. It lowers the barrier to entry to modifying the programs you run, once you grasped the system's basic concepts, dramatically — so much so that I would argue it isn't a difference of degree, but of fundamental principle.

Practitioners of the UNIX philosophy who fear and loathe even warranted complexity should rejoice, too: much of the tower of complexity that has been built on top of UNIX systems that they so like to bemoan — except when it is old and familiar, of course, like the gigantic GCC compiler, or the X Windows System — is the result of UNIX's fundamental tools and abstractions being far too basic, stone knives and bear skins unfit to contend with the modern world, leaving us with no choice but to layer abstraction over abstraction on top of the teetering mess to try to get our systems to understand us and deal with the things we want them to deal with on a first-class basis. With better fundamental abstractions — perhaps reaching back into the hardware itself — even if those core abstractions are higher level and more complex than a core UNIX system, the overall complexity of the system could decrease drastically. Choosing your bedrock correctly matters.

In this respect, this old rant — not unique by any means, UNIX quite often sent PARC and MIT AI Lab and SRI alums who'd used something better into such fits of apoplectic rage they spontaneously became slam poets, as seen in The UNIX Hater's Handbook, although this rant isn't quite of that caliber — is illustrative:

have you ever seen how much work it takes to boot a modern Unix machine and run a C program just to have it print "hello world" in an xterm running under MOTIF? man, it sucks. and it's even more work if it tries to run NT. the machine should be doing a very limited amount of work for this very simple task, but instead it spends minutes booting and preparing itself to be useful, not to mention all the crap necessary to get a program in C able to produce that output. yea, verily, it sucks.

unfair comparison? not at all. why do you think they chose that phrase? because they were developing Unix and the C compiler. it's appropriate to make a machine print "hello world" to verify that everything works after all the mind-boggling nonsense has interfered with the real purpose of a computer, and you never know which part of booting up will fail due to a minor bug. the delight in a C programmer's eyes when his machine thus booted typed "hello world" back at him would probably parallel that of a Common Lisp programmer when the satellite communications subsystem he designed beams back "hello world" after an almost-aborted launch, a navigation jet which misfired, and the solar panels sustained some damage by space debris. normally, it's unnecessary to have confirmations of basic operations, but it makes perfect sense under C.

…

> it's just that unix and windows are set up to support C and C++. e.g., C has a largish libc these days.

these two statements are pretty much contradictory. the problem is that neither Unix nor Windows actually support either C or C++, but they manage to make them work, with downright incredible effort. if you look inside the libraries and see how a system call actually works and how much it differs from the C calling convention and usage, you'd be a fool not to revise your opinion. and does an operating system that forces the programmer to check to see whether the operating system did what it was asked to do every damn time you ask it to do anything actually give any relevant form of support to anyone?

in my view, Unix and Windows support Common Lisp better than they support C because C is designed for a 70's style machine and operating system, which modern machines and operating systems have to mimic with all their flaws and misdesigns, while Common Lisp is a modern language that is well suited to be hosted on modern systems, and it just happens to be, too.

To understand what this poster is complaining about here, you have to understand that because the Lisp Machines they are comparing UNIX to was image based — the state of the system was a living, dynamic thing that which wasn't determined by files on a hard drive, but by the very alive, in progress state in memory, which was then directly to disk, or distributed to other people — and so when you booted a Lisp Machine, it would just flick on instantly, because all it had to do is take the dump of your last sessions live in progress environment and put it back in RAM for you. There wasn't this complex extended bootstrapping process.

You also have to understand that for those machines, there was no distinction between drivers and kernel and operating system and shell and terminal and windowing system and programming language and development environment — it was all one deeply integrated system: not one single gigantic program in the sense that, say, Microsoft Word might be, perhaps, not a gigantic blob of undifferentiated source code, but multiple programs (some ephemeral, like the ones you created in Rep Ls or scratch buffers, some not) so closely interlinked by an dynamic, open system that they became like one program with no barriers between them. They could communicate directly, by calling each other's functions, or directly sharing data structures. Programs, and the operating system itself, were not closed off from everything else, called as a bundle from the outside — they worked more like libraries or frameworks today, where you call up some piece of functionality by directly calling a function that gave you that functionality: even the operating system itself was less a gigantic separate program always running in the background, and more simply a library that the code that ran "on top" of it could call into. There was no need for complex calling conventions and the idea of syscalls.

As another more detailed example: there was no artificial distinction between the "shell" — the Lisp listener, or REPL — and a "terminal emulator," which only communicate via text output, and therefore need some "out of band" way to communicate display information like the horrible teletype escape character hack that is with us today, which then had to be awkwardly herded into displaying richer data structures like images or links using hard-coded special-cases for certain control codes in the terminal emulator software (or, more typically, hard-coded recognition for certain basic patterns like links in the terminal emulator software to make links clickable). Instead, the Lisp listener was both the thing that parsed and evaluated and returned the values of the user's input Lisp commands, and also the thing that accepted the user's input and displayed the output. Thus, Lisp commands could return structured data, with metadata attached about how to display it and how to make it interactive, and the Lisp listener would not only know how to pretty print the data, but could keep that pretty-printed representation linked to the data that created it under the hood, and imbue it with interesting interactive or display properties — like being a clickable button, or an embedded UI widget, or an image — based on the metadata returned, all without the function doing the returning needing to manually initiate anything besides annotating the data, and all without a lossy conversion to some end-state slab of pixels that doesn't remember the data that gave it birth. By this method, images and UI widgets could be truly first-class objects: not merely representable in the language in a nascent data form, to be destructively translated to something else as an output, but with a true identity between what would eventually be shown on screen and interacted with, and the object you "held in your hands" so to speak.

Incidentally, here's a good demo of the basic capabilities of the Lisp listener:

Nothing we can't kind of do a half-broken shallow imitation of today, theoretically, with something like Kitty, but this is just a simple quick demo — it can't really go deep enough into the capabilities of the system to really make obvious what it truly is capable of. The deeper capabilities of this system are far beyond even modern UNIX systems, shells and terminal emulators, which can just barely display images at all, through a horrible hack. If you keep in mind what's really happening under the hood here, however, it should be suitably impressive.

For a more general demo of inspecting a running system program, transparently fetching its code over the network, and then dynamically modifying it at runtime, definitely watch this video as well:

This video is from 1981, so the software demonstrated is several years older than the Symbolics Genera software demoed above, but it is still impressive if you have a little imagination to paper over some of the ergonomics and archaic interface elements. Additionally, at least in this video, Xerox PARC's Interlisp-D, a cousin of the MIT AI Lab's Lisp Machine, comes across as more impressive, but that has more to do with the quality of the presenters themselves in my opinion: besides the structural editing capabilities of SEdit/DEdit, both machines were capable of comparable things when it came to dynamic runtime code modification and the other feats relevant to this essay.

Just imagine what today's computers would be like to use if you could simply pause the execution of any program at any time, see its decomposition at the function level, click on an individual function and have its source code definition immediately called up for you (transparently over the network if necessary), so that you can directly edit that part of the running program, commit the changes, and immediately see them!

So the modern operating system stack looked awkward and clumsy and overcomplicated by comparison: the systems they came from weren't clumsy agglomerations of miscellaneous tiny programs — or even big heaping programs stacked on top of each other, communicating with text streams or serialized data. Their systems were fully interactive, fully integrated, alive. For the Lisp Machines, there was no need for a shell

They would build their programs by just always having the in progress program running, and just dynamically modifying it, redefining things, adding new definitions, inspecting it and stepping through it and responding to errors by directly fixing the error right there and then and then resuming from where the program left off, shaping and molding the program like a living organism or a sculpture until it was done, and then just taking that finished in-memory environment and program and saving it to an image and compressing the image and distributing that as the final program.

This intertwining of development environment and runtime environment may seem unfamiliar, and even odd, to us now, but it actually makes perfect sense. It is essentially what we are clumsily and awkwardly trying to slather on top of UNIX using glorified chroots with containers today: the idea that you develop your program in the same environment it will run in, and then distribute your program with the environment that it expects and was developed for, so that you can always be sure that it will work.

They used this to do other things that we are only now very clumsily and awkwardly and redundantly beginning to emulate with containers, too. There are old textbooks about professional software development with the Smalltalk systems where they describe having a project manager create the image that everyone would boot into in order to develop for the project, so that everyone could develop in the same basic environment, even if they could shape and mold it to meet their needs, and then whenever you made a change or finish the feature, dipping against That Base image and sending the diff back. Because that's right — they actually had image-based version control, apparently, and apparently it worked really well.

So in the end, by choosing a better set of bedrock abstractions and paying a little complexity up front, a modern Lisp Machine could avoid the piles and piles of useless, obfuscatory, obstructing abstractions we needed to place on top of UNIX just to adapt a piece of software that was designed to run a paper teletype from a PDP-11 work in the modern world.

Even if you don't believe me on that, consider this, too: simpler may be better, all else being equal, because it's easier to understand and there are fewer points of failure, but again, this isn't a question of absolutes (at least not in today's world, where hard hardware limits aren't as much of an issue) this is a question of tradeoffs, ratios, and all else isn't equal: the simplicity of UNIX makes it terrible to use and almost unethical, in comparison to what we could have, while the slightly greater complexity of a Lisp system gives us the four freedoms on a silver platter and would be infinitely better and more useful. I think, personally, that is a worthy tradeoff.

Where did I get that glimpse I mentioned earlier of what a non-UNIX world could look like, and how much better it was? What showed me that it was the UNIXisms of the modern free software ecosystem that were holding it back from reaching its true apotheosis, maximum user control and autonomy? Really, that realization was spurred by two things: first, on the negative side, were my experiences with *nix system administration, monkeying around in the pipes and wires of things like Gentoo, FreeBSD, and regular GNU/systemd/Linux (systemd is quote good, by the way), as well as finally diving into the technical details of UNIX, crucially, far away from the cultish echo chambers of the UNIX partisans that eternally try to convince each other there is nothing better. But second and more importantly, on the positive side, the thing that showed me not just that UNIX was flawed, but at least one possible, beautiful, wonderful way out of that morass, was the text editor Emacs.

What? How does that make sense? UNIX is an operating system, and Emacs is merely a text editor!

Well, I would argue Emacs is more than that. It is a computing environment, a general purpose interface through which you can interact with and control your computer and perform various tasks. It may sit on top of another operating system, yes, but it acts very much in the way you might want the userland layers of a truly integrated, high-level operating system to act, and so it can teach us lessons about the design of operating systems. You see, to understand what Emacs truly is, we should look at what parts of it are truly immutable, bedrock abstractions. And the only immutable part of Emacs — incidentally the only UNIX-y part, written in C — is the interpreter, compiler, and bytecode VM of the Emacs Lisp language, parts of its standard library, and the advanced Lisp Machine-inspired display system integrated into that runtime. The editor part of Emacs is all written in Emacs Lisp itself, running on top of the tiny core known as temacs, and in theory could be completely removed or drastically modified — and in fact, if you run Emacs in "batch" mode, that is sort of what happens — so it is more like Emacs the editor is merely the "killer app" or vendor-provided example program of Emacs the Lisp environment. Thus, we can say that GNU Emacs is, in this sense, the Last Lisp Machine. And indeed, many of those far more well versed in the esoterica of the Church of Emacs than I speak of Emacs in this fashion.

Emacs-as-Lisp-Machine is very interesting, too. It is the one dinosaur that survived the extinction event that was the victory of Windows and UNIX by, in this analogy, "becoming a bird": rather than digging ever deeper into its isolated hole like the Smalltalk environments did, it survived by adapting to its new environment: learning to deeply understand, control, and integrate with the new things in its ecosystem — UNIX files and directories, text, regular expressions, processes, and more — and to occupy a relatively stable ecological niche within that ecosystem — as a text editor — while still maintaining its fundamental DNA, that of a dynamic, shapable, fully-integrated high level environment.

I won't claim this was fully intentional, but I do believe it is the end result. After all, when you are running Emacs, you will find that you have at your disposal a Lisp language and runtime deeply integrated with a highly interactive¹ general-purpose computing environment and its powerful user interface engine, almost all of which is written entirely in that same language, and designed to take advantage of that deep integration both to be the best possible development environment for that language, and also to allow that language to dynamically inspect, modify, mold, and shape that environment interactively, at runtime, like a living organism, even as the environment shapes and molds the programs shaping and molding it, much like the original Lisp Machines.

And once again like the Lisp or Smalltalk Machines, Emacs acts as a language VM that is here not simply an invisible interpreter in the background, but an environment you can enter and explore and live in and control and extend and shape and mold through the language it interprets, and which understands language and interface concepts at a seamless first-class level, in a way an external OS around a VM system never could, but which also grants you powerful abilities to interact with the outside world.

And finally, also like those Lisp Machines of yore, Emacs too is designed with a close correspondence between the actively running program and the source code that defines it, the ability to dump images of its runtime state, and an extensive and powerful hypertext documentation system that can show you extensive documentation or the source code for any and every part of the system. All of this is just running on top of a UNIX system.

All of this is merely running atop a UNIX operating system, instead of being integrated top to bottom.

Oddly, in a way I think seeing and using Emacs this way is a fulfillment of the original dream of the GNU project, too — even though I think ultimately the choice to base the GNU Project on UNIX, while a smart technical move at the time (UNIX was ascendant for a reason), has ultimately become baggage holding it back. Recall for a moment the original GNU announcement:

GNU will be able to run Unix programs, but will not be identical to Unix. We will make all improvements that are convenient, based on our experience with other operating systems. In particular, we plan to have longer filenames, file version numbers, a crashproof file system, filename completion perhaps, terminal-independent display support, and eventually a Lisp-based window system through which several Lisp programs and ordinary Unix programs can share a screen.

Or Stallman's recollections of the origins of Emacs, quoted above:

At first, I thought of making a Lisp-based system, but … I decided instead to make a Unix-like operating system that would have Lisp implementations to run as user programs. The kernel wouldn't be written in Lisp, but we'd have Lisp. So the development of that operating system, the GNU operating system, is what led me to write the GNU Emacs.

Or the fact that Emacs is even what Stallman still largely spends all his time in:

Mostly I use a text console, for convenience's sake. Most of my work is editing text and that is more efficient on a text console. … I spend most of my time editing in Emacs. I read and send mail with Emacs using M-x rmail and C-x m. I have no experience with any other email client programs. … I edit the pages on this site with Emacs also … I have no experience with other ways of maintaining web sites.

In my reading of these quotes, it strongly implies that Emacs was the environment Stallman wrote for himself in lieu of the Lisp-based operating system and windowing system he wished he could have written, but settled on UNIX instead for.

Emacs it really is a general-purpose computing environment, too. What else could you call an environment that is general enough to express many of the applications one might want (the ones that are mostly centered around text and images, anyway) within itself, including a basic web browser, a mail sending and reading system, an RSS reader, a calendar and diary, a powerful literate programming/Jupyter notebook, note taking, and personal planner system, an Obsidian-style Zettelkasten application, a terminal emulator, a Lisp Listener-like system shell not distinct from its own terminal emulator, a windowing system (of sorts, or a full one), a powerful graphical file manager, a document viewer, a multimedia manager, an entire VIM clone, a customer service system, an air traffic control system… and of course a little-known but very powerful text editor? And importantly, it expresses all of these in such a way that every application, while a complete application and program in itself, also functions as an open library of functions, commands, and utilities that you can reach into and use for your own purposes directly — thus, every command or piece of functionality of those applications becomes fluidly, easily scriptable, without the need to break things artificially into little chunks — the chunks exist already, since every program is open to dynamic inspection and modification by the Emacs environment.

On the subject of the Emacs display system, by the way, how does it work?

Well, for instance, I recently decided to switch dashboard packages to something faster, but it didn't implement image banners by default — so what did I do? The new dashboard package obviously lets you pass in structured data (lists and plists) to decide the structure of your dashboard, where there are strings that it uses to print out the menu items, but how could I extend this to images? Well, it turned out to be extremely simple: I just had to make the headline menu item a string with metadata attached to tell Emacs's display system to display an image on top of it. That's right — images are first-class objects in Emacs, and all you need to do to display them is attach them as metadata to a string. Note, this doesn't produce a new kind of string object that some parts of the language or editor know how to interpret and some don't, and when it is output, you don't lose the text or the metadata — it's just a regular string, imbued directly with rich multimedia metadata, that can be treated like a regular string by your code, while also showing up as an image in the display, from which the original text or metadata can be derived. Here's the first part of the code:

(concat
          "    "
          (propertize "QUAKE EMACS" 'display (create-image "~/.emacs.d/banner-quake.png"))
          "\n"
          "\n"
          (propertize  (format "Started in %s\n" (emacs-init-time)) 'face '(:inherit 'font-lock-comment-face)) ...)

That's right, you can just use the regular string concat function to concatenate together images and text, because images are just text with metadata attached! Here is how it looks:

And this is how the entire UI works: it's all just structuered data where the leaves are texted annotated with metadata to control its display and interactivity. This metadata principle is how you make text taller, or turn text into buttons and attach handlers to those buttons or draw images or different fonts or icons or display arbitrarily-shaped and sized GUI elements using SVG or embed GTK+ widgets — the entire interface is just structured lists of text with metadata attached that turns that text into rich UI elements in a predictable and deterministic way; and you can get the original text and associated metadata that was used to construct a part of the user interface back out of it at any time, just from the UI itself. Combined with the fact that you can attach arbitrary metadata to that structure text — and in fact that Emacs programs often do so — and also tell the display engine how to interpret new types of metadata, and you have an extremely rich way of building introspectable interfaces! And in the end, you can just use the regular string or list processing functions on all of it, buttons, images, various sizes and fonts of text and all, because images are just text with metadata attached.

This system is even more dynamic and introspectable than HTML is — especially since you can attach arbitrary metadata, so it is common practice to attach the relevant internal UI state as metadata to each UI element you push out to the screen, meaning that any program can reconstruct the state that generated any UI that it is looking at. Not the deep global application state of course, but the stuff conceptually "right behind" the UI anyway. The power of having the end-product user interface your program generates not only be generated from native language data structures and constructs, but also be derivable from that generated UI, so that the native language nature of it all is bidirectional, is incredibly useful, almost mind-bogglingly so to me. HTML/the DOM/JS gets close to this — that's a huge part of why the web stack is so fundamentally powerful and popular — but in my opinion it is a bloated, poorly designed in many respects, version of this, not intended for user interfaces initially; in my opinion, it is also less powerful and interesting, partly because HTML and the DOM aren't very native to JS in actual fact, and partly because with HTML, determining how to interact with and display content comes prior to the content, instead of being attached after.

Yet, the best part is, it isn't just a dumb overlay, or an image broken into little character-size pieces, like in terminals that can "display images": this is a full single image (jpg, png, svg, what have you), rendered as a single image, with the text around it flowed to allow it to fit properly, just like on the DOM (although the reflow algorithm is a lot more predictible imo: the image is a single "logical line", the line is just bigger). The best part is — if you're in an emacs lisp REPL, like eshell, where the data structures returned from the expressions you are evaluated are pretty printed…

yup that's right, native support for return values that are images! You can even wrap this in a command and use it to cat images. Here's how you'd dynamically extend the built in Emacs Lisp shell's cat command to be able to cat images and Emacs buffers as well as files in 13 lines of code:

(advice-add 'eshell/cat
            :around (lambda (oldfun &rest args)
                        (mapconcat (lambda (arg)
                                       (cond
                                        ((bufferp arg) (concat
                                                        (with-current-buffer arg (buffer-string))
                                                        "\n"))
                                        ((string-match-p (image-file-name-regexp) arg)
                                         (concat
                                          (propertize "image" 'display (create-image arg))
                                          "\n"))
                                        (t (apply oldfun args))))
                                   args)))

And here's me creating and returning a button object, which is then pretty-printed as a button by the REPL (this isn't me inserting the button, I'm just returning the button object and it's displaying it) with a callback that runs when clicked:

(You could give the buttons outlines and depth and stuff, but I don't bother.)

In a lot of ways, it's like having the stuff iPython Notebook gives you, but as a more powerful, generalized system you can build whole interfaces with instead of it just being a special-case of a REPL that can render return values — it's a whole rendering engine that works based on the metadata of objects passed into it, and you can combine them or add your own.

And all of this is simply how the Lisp Machines worked to produce their entire display.

What all of this means, in a nutshell, is that that the experience of using Emacs as an extensible computing environment is as close as one can get, these days, to using a Lisp Machine. To using a system under which the ability to inspect and modify the code (and thus, more importantly, the behavior) of a program you are running on your computer is truly maximized. And I cannot begin to describe the feeling of freedom and self-sufficiency and confidence that it brings to you — one that hardcore Gentoo or BSD users can only dream of, because while they can awkwardly duct-tape together various largely black-box programs broken at artificial places into little pieces, or tweak configuration files hoping that their use-case has been predicted by the creators of those programs in some combination of configuration parameters, or painstakingly and exhaustingly download, modify, compile, and patch in a new version of whatever tool they want to change, without being able to see where its running behavior or expressions correspond with its code, for me it is trivially easy to pop open a scratch buffer, write a bit of Lisp to modify or redefine some behavior or add some command in some program in Emacs, evaluate it, and continue on with using it, or find a command and jump to its definition to edit it. The program shapes and molds itself to me in a way that really makes me understand what those four freedoms mean.

In the end, what am I trying to say? Well, try Emacs out for one. You'll probably hate it — it is 40 year old software, the composition of all the use cases and tools that its users and maintainers felt they needed over the years (although I absolutely love big, well rounded, full-featured, complete software like this), using idiosyncratic and outdated jargon, and with practically fractal learning curve — but if just the right thing is broken in your brain, maybe you'll fall in love with it like I did.

More importantly though, what I think I'm trying to issue here is a plea to free software lovers everywhere to think outside the box a little. There are more places in the operating systems design space that we can explore than just the UNIX space (Linux, BSD, Plan 9, etc), more historical (or even contemporary) operating systems we can steal good ideas from, and new avenues to explore with greenfield projects instead of simply recapitulating "UNIX but slightly better" over and over again (like Redox OS, Serenity OS, etc). We need to be willing to ruthlessly steal good ideas from anywhere — stop dismissing ideas simply because they are from Windows or Mac, or simply because they aren't sufficiently in line with tradition or the "UNIX philosophy."

Furthermore, we need to avoid falling into the trap of dichotomous thinking. The proponents of UNIX (or the "worse is better" methodology in general) like to frame things as a strict binary dichotomy between, spiritually if not literally, MULTICS on the one hand and UNIX on the other — either getting lost in eternally striving for unachievable perfection and never actually getting anything done, or simply giving up and going for whatever is simplest and solves 80% of the basic problem (without paying attention to anything else). Many will argue that we must stop trying to achieve anything significantly better than what is easy and already exists, stop critiquing what actually exists so that it might become better, or so that people might build better things in the future, in the name of productivity: that we should simply give up and accept simply glomming more slop onto the structure that tradition has bequeathed us without stepping back to actually try to (re)design or improve anything, or else we won't get anything done. But this is just the superstitious reification of a historical accident into a way of thinking, not a serious philosophical point: UNIX itself was built as a conscious reaction to the disaster that was designing and attempting to build MULTICS, and, even though that was a particular historical circumstance that is not generally applicable everywhere, much of software development ever since, having been influenced by UNIX's post hoc "philosophy", is also an unknowing reaction to long-dead MULTICS.

This is a false dichotomy, though: we can try to find better ways of designing and building the systems we use without letting perfect be the enemy of good enough, even if in doing so we reject some first principles of the older systems, or have to do a little first principles thinking of our own. Trying to come up with a fundamentally better system for doing something than what is used now is not a "Poetteringism" to be tossed away as evil simply because it changes things. It is the path to a brighter future. Because "Worse" and "The Right Thing" exist on a spectrum, and with a little thinking and a lot of effort, we can try to do the best we can to move the needle a little more towards good things, while still acknowledging and adapting to existing constraints and trying to actually deliver something. After all, the Lisp Machines and their D-machine cousins did actually come to fruition, were actually sold and delivered to market — they didn't lose because of some fundamental philosophical flaw like an obsessive perfectionism preventing them from even reaching the market, they simply lost because they were too ahead of their times and hardware had not yet caught up to software. In fact, the whole ideal of gradual, organic, iterative development, where you get a prototype of your idea working, just enough so people can get the idea and maybe use it, and then organically evolve it from there — the very opposite of the kind of "Right Thing" mentality that might stop something from coming to market — was at the heart of the systems I'm talking about. And yes, I am all for not letting the perfect be the enemy of the good enough, but what we have done, and what we are often doing, in the software industry is amazingly the opposite: choosing the dumb, ad hoc, hacky solutions that paper over the cracks in our fundamental ideas and abstractions, instead of choosing to use just a little bit of all our collective intelligence to make things better in any real way.

So I'm not saying we need to design the perfect computer system up front. I'm saying that now that we have more powerful computer hardware and the benefit of hindsight, we can make things — not perfect, but much better than they are now, and that we must seriously consider doing so, because what we have now, while it theoretically aligns with our values, does not make access to them readily or widely available and accessible, and therefore renders them largely theoretical. I'm saying that we need to really carefully analyze how our values, in the software domain as in anywhere else, actually work out in practice in the current paradigm: whether, in the real world, the things we believe should hold true, and say hold true in theory, actually hold true practically, for most people, in most situations, using our software. And then we need to carefully consider what other avenues we could pursue that would serve our ends better.

There's more to be seen out there, more to do, more to design.

There's new OSs to write.