Unique aspects of GNU Emacs

Unique aspects of GNU Emacs   emacs

A general computational environment integrated from inside and outside

To understand what Emacs is, you can look at it in two ways:

  1. The fundamental tool set it provides to users (for scripting and configuration) and package developers. If it was just an editor, that tool set would be limited to a scripting language with access to an API for controlling a pre-made editor interface, and a few minimal ways to connect with or control the outside world. That's because something that's fundamentally a text editor isn't going to be interested in exposing the core command loop, the fundamental building blocks the editor uses for building its user interface, the ability to respond to arbitrary key presses (so non-editor-like UIs can easily be built) and powerful APIs for interacting with the host system. Why would you need that kind of access?
  2. The parts of Emacs that can't be changed from inside it. That would be the C core, representing currently around 25% of the total code base. If Emacs was just a text editor, the unchangeable core would be composed not of general purpose systems that could be used by any kind of text-oriented computing environment, but systems implementing a specific text editor's user interface and functionality.

Neither of these things is actually the case, however. If you look the C core and fundamental tool set provided to package authors and users by Emacs, what you see is:

The rest is all purely in Emacs Lisp – open to be changed, adapted, modified, or overridden completely.

This doesn't look like the core of something that is essentially a text editor. It provides way too much:

  • the general purpose building blocks for creating basically any kind of rich text-oriented interactive interface with its command loop, keymap system, and display system;
  • a standard library that is extremely good at managing processes, ingesting and transforming not only text but also structured data, and doing inter process communication (so good that it even begins to rival Python's standard library);
  • a system that provides a common set of user interface elements (in the form of the echo line, mode line for each buffer, and window management system) for arranging and managing the multiple user interfaces and applications in a coherent and consistent single user interface environment;
  • an environment where all of these interfaces/programs can communicate with each other, call each other's functions, and even inspect each other's user interfaces, thanks to being in the same environment courtesy of the previous point.

This deep integration into a coherent computing environment allows for some truly incredibly impressive automated and integrated workflows, like those of Sacha Chua: see here Audio Brain-Dump Workflow and the way she runs EmacsConf.

This is why, in my humble opinion, the old joke that "Emacs is a great OS" is true. It is a general purpose, Lisp-based single address-space, homogeneous, malleable system designed for ingesting, transforming, and operating on textual and structured data in an interactive text-oriented user interface environment.

Unlike many other homogeneous, malleable computing environments surviving from the Before Times – such as Smalltalk – however, Emacs hasn't become increasingly sealed away in its own little hermetic world. Instead, thanks to its powerful filesystem and process management capabilities it can leverage the world of composable Unix command line tools, much like a shell does, and it can use its powerful IPC and HTTP abilities to talk to non-command-line services as well. It thus regularly builds user interfaces for external tools and services such as Magit, can read and recognize the output formats of common Unix tools like make, ls, diff, grep, and GDB and turn theme into interactive (and even editable) hypertext, often is used to control other applications, and contains powerful capabilities for communicating with and managing its host system (see also). It even has a full cross-platform, native (Lisp-based) POSIX-like shell built into it. Emacs is a Lisp Machine adapted for a world where UNIX won.

Emacs really is a general-purpose computing environment, too. What else could you call an environment that is general enough to express many of the applications one might want outside of anything related to code editing (the ones that are mostly centered around text and images, anyway) within itself, including:

and of course a little-known but powerful text editor?

In a sense, the text editor itself just falls naturally out of the text-oriented computing environment toolbox, and to the degree that things are built on top of that to foster an actually excellent and usable text editor, instead of just a set of components to make a text editor, well, that text editor can simply be considered the "example application" of this powerful computing environment!

Why are people so obsessed with living in this computing environment, though? That's down to the other unique aspects, that all add up to one word: freedom.

Transparent from the inside

One of the foundational things about Emacs that makes it unique is that every single function and variable that makes Emacs up, barring the C core (which only makes up about 25% of the codebase, and which everyone is always looking to shrink if possible) whether or not it is intended for the user to run as an interactive command, or even for anyone to look at or reuse at all, is transparently there, available, at all times, for the user to inspect and run to see what happens, or recompose into new functionality, or call to script or automate tasks. There is absolutely no limit. There's no small defined set of API functions you can call and variables you can read, no boundaries within the system, either to you as a user, or to your code, or to the other packages you install.

This means that absolutely every piece of code writtenf or Emacs is maximally reusable.

Not only that, but every single one of these functions can not only be run but carries its own documentation and full, original source code (gzipped) along with it so that the user can fully understand what every single thing in the system does and how it does it.

It's also really important to consider what the Emacs computing environment does with this transparency and the fact that documentation can be attached to every symbol in the editor. For one thing, while many programming languages now also have the concept of attaching documentation strings to functions (for example, Python), few allow those docstrings to be rich markup with hypertext capabilities, especially not hypertext capabilities to link to other language symbols and datatypes. Even fewer languages provide their complete introductory, reference, and user manual as on-line (as in, provided with, and accessible at a granular level from within, the software product) marked up hypertext documentation, along with full user manuals for every built in package and library.

It is even rarer for application systems to expose this style of on-line granular (per-functionality) documentation, in addition to user manuals and reference manuals, for their actual user interface and functionality, instead of it just being an internal developer convenience. Emacs is one of the very rare applications where you can quite literally ask it for a detailed description of what any button (command) or key combination (key chord) does, with links to the reference manuals, instead of having to search manually through the user manual hoping for a section that specifically covers the button you're looking for (and it may not, since user manuals are usually much too high level to cover in detail the functionality of each individual button).

Still fewer systems provide so many systems for accessing this documentation: if you press C-h C-h (the help system's help menu), you get this insane list of ways to access the help system:

Commands, Keys and Functions

   m    Show help for current major and minor modes and their commands
   b    Show all key bindings
   M-x describe-key     Show help for key
   c    Show help for key briefly
   w    Show which key runs a specific command

   a    Search for commands (see also M-x apropos)
   d    Search documentation of functions, variables, and other items
   M-x describe-command     Show help for command
   M-x describe-function    Show help for function
   M-x describe-variable    Show help for variable
   o    Show help for function or variable

Manuals

   r    Show Emacs manual
   F    Show Emacs manual section for command
   K    Show Emacs manual section for a key sequence
   i    Show all installed manuals
   R    Show a specific manual
   S    Show description of symbol in pertinent manual

Other Help Commands

   C-e  Extending Emacs with external packages
   p    Search for Emacs packages (see also M-x list-packages)
   P    Describe a specific Emacs package

   t    Start the Emacs tutorial
   C-q  Display the quick help buffer.
   e    Show recent messages (from echo area)
   l    Show last 300 input keystrokes (lossage)
   .    Show local help at point

Miscellaneous

   C-a  About Emacs
   C-f  Emacs FAQ
   C-n  News of recent changes
   C-p  Known problems
   C-d  Debugging Emacs

   g    About the GNU project
   C-c  Emacs copying permission (GNU General Public License)
   C-o  Emacs ordering and distribution information
   C-m  Order printed manuals
   C-t  Emacs TODO
   C-w  Information on absence of warranty

Internationalization and Coding Systems

   I    Describe input method
   C    Describe coding system
   L    Describe language environment
   s    Show current syntax table
   h    Display the HELLO file illustrating various scripts

This isn't even all of them. In what other user interface system can you run a command that means "tell me what internal function what I'm about to do does", then right click on something, navigate through a menu, click on a final command, and see that function, then trivially jump to its definition and inspect it, or get its documentation, as demonstrated here?

Possibly the coolest help commands in the Emacs computing environment are the apropos family of commands – that's right, there's not just the one! Using C-h a apropos to get a list of commands containing the word apropos, we see:

  Type RET on an entry to view its full documentation.

apropos                       <menu-bar> <help-menu> <search-documentation> <find-any-object-by-name>
  Show all meaningful Lisp symbols whose names match PATTERN.
apropos-command               <menu-bar> <help-menu> <search-documentation> <find-commands-by-name>, <help> a, C-h a
  Show commands (interactively callable functions) that match PATTERN.
apropos-documentation         <menu-bar> <help-menu> <search-documentation> <search-documentation-strings>, <help> d, C-h d
  Show symbols whose documentation contains matches for PATTERN.
apropos-follow                M-x ... RET
  Invokes any button at point, otherwise invokes the nearest label
  button.
apropos-function              M-x ... RET
  Show functions that match PATTERN.
apropos-library               M-x ... RET
  List the variables and functions defined by library FILE.
apropos-local-value           M-x ... RET
  Show buffer-local variables whose values match PATTERN.
apropos-local-variable        M-x ... RET
  Show buffer-local variables that match PATTERN.
apropos-user-option           <menu-bar> <help-menu> <search-documentation> <find-options-by-name>
  Show user options that match PATTERN.
apropos-value                 <menu-bar> <help-menu> <search-documentation> <find-option-by-value>
  Show all symbols whose value's printed representation matches
  PATTERN.
apropos-variable              M-x ... RET
  Show variables that match PATTERN.
command-apropos               M-x ... RET
  Show commands (interactively callable functions) that match PATTERN.
customize-apropos             <menu-bar> <options> <customize> <customize-apropos>
  Customize loaded options, faces and groups matching PATTERN.
customize-apropos-faces       <menu-bar> <options> <customize> <customize-apropos-faces>
  Customize all loaded faces matching REGEXP.
customize-apropos-groups      M-x ... RET
  Customize all loaded groups matching REGEXP.
customize-apropos-options     <menu-bar> <options> <customize> <customize-apropos-options>
  Customize all loaded customizable options matching REGEXP.
gnus-group-apropos            M-x ... RET
  List all newsgroups that have names that match a regexp.
gnus-group-description-apropos M-x ... RET
  List all newsgroups that have names or descriptions that match
  REGEXP.
info-apropos                  <menu-bar> <help-menu> <more-manuals> <lookup-subject-in-all-manuals>
  Search indices of all known Info files on your system for STRING.
tags-apropos                  M-x ... RET
  Display list of all tags in tags table REGEXP matches.
xref-find-apropos             <menu-bar> <edit> <goto> <xref-apropos>, C-M-.
  Find all meaningful symbols that match PATTERN.

We've got everything from full-text documentation search to full-text reference manual search, and every kind of specific search you need in between.

This kind of variety and power in help functions is important because a user that needs help finding or understanding something is always going to know a different subset of the information about the thing they want to find, so you need a command for each possible set of information they might already know that can use that information to find the information they don't know. If you only provided help that goes in one direction, or deals with one cross-section of information, than users whose knowledge doesn't happen to match up with those assumptions are going to be left completely lost.

Malleable at run time

From inside Emacs, while it is running, you can:

  • add new code
  • evaluate expressions
  • redefine functions, variables, or data types
  • create new commands

and all of these changes will take effect immediately, modifying your running system live as you use it, from within it: using Emacs to modify itself. You can also use this functionality to inspect and analyze the state of the running system, as it runs – a feature extended by the Emacs Lisp debugging system, which allows you to have the full power of a debugger as you're using and modifying your running Emacs.

This sort of live, self-referential experience is simply unmatched by most computing environments. There's really nothing like it. It's somewhat similar to the concept of code hot reloading that many web developers may be familiar with, but infinitely more granular, since you're just sending code and new definitions directly to the running system, no need to save whole files or modules, and without the need to totally reload a program and then recreate its state – instead the program just changes without you needing to do anything. Moreover, even hot reloading is not applied, to my knowledge, to systems modifying themselves, instead of just updating other programs.

In fact, the entire way you configure Emacs is built on the back of this run-time malleability: instead of a static configuration file in a dead data format like JSON or TOML that the editor reads and then uses to internally set configuration values and add commands itself, the configuration format of Emacs is Emacs Lisp: when you find a modification that you like through changing Emacs at runtime, and decide that you want it to apply to future Emacs sessions, you just copy those Lisp forms into Emacs's init.el file, which is literally just a file of Lisp code that's run every time Emacs starts up. And eventually, if your customizations eventually become complex, powerful, or interesting enough, you can trivially move them out into a package that anyone else can install, without having to transition from a configuration format to whatever language extensions are written in like other languages. There's no barrier between modifying the editor at runtime, more robustly changing it for yourself, and sharing those modifications.

Modifications compose robustly

Since redefining things is a built-in language feature, not a meta-level feature of some build system or hack, like "hot reloading" is, the redefinition of a function, variable, or data type can be done without modifying the original code: simply evaluate the new definition, and it will replace the old one in the running state of the system (the "image" in Lisp parlance) without corrupting or modifying the original definition, safely stored in source code form on disk, in any way. Thus, you can have the Lisp code in your init file programmatically perform the redefinition at startup time, and you can have the benefits of persistently modifying some aspect of your system, without having to deal with merge conflicts down the road when new versions of whatever you were modifying come out, or having to worry about losing the original version should you realize there's something wrong with your modification and want to compare or revert it.

If you don't want to completely replace the definition of something, but instead simply want to modify its general behavior in some way, you also have access to the ability to "advise" functions in Emacs Lisp. This functionality allows you to modify the inputs to a function, the outputs from a function, run the function conditionally, run other functions always before or after that function, and any one of a myriad other options, all without modifying or even redefining the original function. Importantly, as well, each of the ways that you can modify the behavior of such a function is represented as a stack of such modification lambdas, thus allowing multiple pieces of code that have absolutely no clue as to the presence of the others to modify a function composably, with a much lower chance of disaster than if they were simply wholesale redefining functions. (Of course, this isn't perfect, but it's an extremely powerful tool).

Unity of internals and user interactions

  • Users edit text by using a suite of powerful text navigation and manipulation commands on substrates for holding and displaying propertized (semantic+structural metadata infused) text (buffers). This is also how Emacs Lisp internally performs all of its text manipulation and UI building. There's no artificial separation between the two, which means that tools and functionality that help the one can be available to help the other, and vice-versa. See also.
  • Users control the program by pressing keys which are directly mapped to functions that run within the current running process, or through using execute-extended-command. The inputs they are interactively prompted to provide to those functions are just regular arguments, and they can also use key commands to supply such arguments prior to activating the function, as well. "Interactive commands" are literally just Lisp functions, only separate from functions for convenience so that when users are trying to run commands they don't get suggestions for a bunch of functions not usually relevant to interactive uses, and so that those functions can specify how to interactively prompt for values for their arguments. That's it. Users can call non-interactive functions interactively, and non-interactive Lisp can call interactive functions transparently.
  • User interfaces are presented to the user the same way they're represented internally, and the same way user-editable content is: as just propertized text in buffers. The user can always toggle off read-only mode and edit it, or run another application on the text provided as an interface by the previous, or copy the text out to a new buffer for inspection, or do anything else they want with it, just like the program that created the text UI does. This also extends to Lisp other than the Lisp that created the UI, too. Moreover, since buffer manipulation is the core of Emacs Lisp, its absolutely most powerful capability, this means that all of this power is available for all user interfaces. Everything is a buffer, and so everything can have the full power of Emacs brought to bear on it. Thus every UI is scriptable, inspectable, readable, modifiable, by everything within the system, instead of being a flat black box. A buffer is a buffer is a buffer.

This also allows some really cool things, like command-history, which shows the history of commands the user has run as executable Lisp code. Note that this is in part made even possible in the first place because in Lisp, code is just data, so it's easy to return a value that represents the code that was executed without creating a special command language and interpreter thereof, like you would have to do if you had used, for example, JavaScript. A practical application of this is the ability to export a keyboard macro as a Lisp function, for addition to your config and use later, or just out of curiosity, and have the code produced thereby be a first class citizen, operating in exactly the same way "real" Emacs Lisp code should.

Infinitely malleable

Thanks to the previous points, Emacs can be said to be truly infinitely malleable. Want to replace a core feature, in such a way that everything using that feature immediately and transparently uses the new version? Sure thing, that's trivial – common, in fact. See: Vertico, Corfu, Ido, IComplete, etc. Want to use any part of the existing editor yourself for something else? Sure, you can do that too. Want to change how a core command behaves to fix a bug or improve a feature? Certainly. Want to define new commands? That's so easy it's barely remarked upon.

This is in stark contrast to even editors like Visual Studio Code – which is often touted as being Emacs's equal in terms of customization – but which:

  1. makes a strong distinction between customizing the editor (through flat data and a GUI) and creating an editor extension,
  2. forces you have to make a painful transition from its customization interface to creating a full on extension with a preset project format and a lot of boilerplate and a complex API even for adding a simple command (leading to users asking VSCode developers to reinvent a mini command programming language in JSON to regain the ability to easily write macros),
  3. makes you customize your editor through the pinhole of a powerful, but still rigidly circumscribed extension API, instead of giving you full access to everything (meaning you can't fundamentally change the behavior, layout, UI, or editing functionality of the editor),
  4. due to the previous point, while Visual Studio Code may seem to offer more powerful UI capabilities than Emacs due to being based on a fully featured web browser through Electron, thus allowing you to embed full web applications inside it, these GUI extensions will be unable to use the same UI features and paradigms as Visual Studio Code (such as their own vertical fuzzy searching palettes) since the API doesn't expose the internal tools necessary to do so, and the rest of the editor will be unable to be aware of and thus unable to integrate with or use anything in the UI, thanks in part to extensions not being able to read the editor's user interface (unlike Emacs), and the core of VSCode being unable to be extended to know about the web UI that's been shoved into a tab.

Thus you can see that if you want true malleability in your computing environment, what you want remains – and probably will always remain – Emacs.

Why would you want that infinite malleability? Because it allows you and the software you use to grow together:

Emacs is the real experience in digital gardening. You tend to it for years, decades, as it grows with you to be your own, one-of-a-kind computing environment where you feel comfortable, efficient, and in peace.

It's not as perfect as the commercial flower installations. But they come and go, and your garden endures. As new plants come into fashion, you can get them for your garden too. As they go out of fashion, you can still keep them if you want.

And yes, you can also blog with it too.

Any open, cross- environment where you can change, compose, automate, and adapt any of the vast panoply of tools and interfaces it offers you to fit how you think and how you want to work, instead of only having a few toggles here and there and otherwise being forced to hope you can find a near-enough fit in rigid existing tools or adapt yourself to the workflow of some tool or other, is an environment that is worth investing time into, because it can truly adapt to you and become yours. Such a piece of software is well-worth adopting as personal infrastructure, making you more productive and happy in small ways, letting you move through your tasks with greater ease.

Application output ("UIs") are composable and open to reflection

Thanks to the fact that interfaces in Emacs are not just:

  1. highly structured trees of structured data, which can be unwieldy and rigid to transform,
  2. nor merely the blank slates of pixel buffers,
  3. nor just text, which is an overly-flexible communication medium where semantic meaning and structure is easily lost, especially when there is no out-of-band information and the text must be structured into a UI that is pleasant for humans to look at,

but a combination of all three (propertized text, where text can be propertized with images and such), interfaces in Emacs are special in several ways:

  1. The same interface can be represented in multiple different ways: <https://lists.gnu.org/archive/html/emacs-devel/2020-09/msg00286.html>
  2. It is trivial for different programs and pieces of code to inspect, search, modify, manipulate, and reinterpret the raw content (the text) of the UI in whatever way it wants, so that you can essentially take the UI created by one program as its output and input it into another program when you're done using the UI as intended, or have multiple programs doing different things with the same UI content at once if you want to (like using Embark on a Magit line).
  3. Yet at the same time, the semantic meaning of the data represented by the text layout is not lost, nor do ad-hoc structured data formats need to be created to preserve that meaning while remaining human-usable for every program, like in the Unix shell, because a common set of structured data types are understood by all Emacs Lisp code, and subsections of any text in any buffer can be annotated ("propertized") with that data as metadata, to document relations between different textual elements and the semantic meaning of them. Thus, inspection can use just the text level to radically reinterpret or modify UIs, or inspect the properties of that text to respect and integrate with the original intentions of whatever code created the UI without needing access to the internal state of that original code.
  4. Plus, we're not just limited to static, boring old plain text! Since the display system can also read those properties, special properties can be used to go beyond just plain text, including rich text things like font sizes, styles, and faces, as well as images, PDFs, and SVGs – but since all these things can only be introduced as annotations on text, there must be a text alternative to each underneath to be annotated in the first place, thus encouraging the use of text as a substrate and therefore reinforcing all the previous points!

Essentially, propertized text is Emacs's thin waist, allowing extreme composability and flexibility within a common structure and set of primitives.

Thus for example we can have a tool like Embark, that can use advanced patterns to examine the raw text around the cursor when it is activated to determine what kind of "thing" is there and what can be done on it, but which can also inspect the structured metadata attached to that text to more intelligently figure out the "thing" and available actions, so that for instance, when you run Embark on the fuzzy selection UI of Emacs itself (since all UIs in Emacs, even the system level, core ones, are just propertized text), it can read the metadata of each line of the UI to see which are the input and which are the suggestions, and what the suggestions are (buffers, files, etc), and which one the user has selected. It can even export the completion interface as it currently exists to a new buffer for you to preserve as a menu, edit, or manipulate, with all the semantic meaning, button functionality, and even styling preserved. And this can all be done without having to query anything internal, just by doing runtime reflection/introspection on the UI.

A powerful set of limited primitives

One of the most important things about Emacs is that, as flexible as it is, it is not infinitely flexible.Everything operates on the basis of a common set of interoperable building blocks that are themselves very flexible, but not too flexible – it is important that it is still a fully-unified homogeneous environment:

  • everything displays and operates within a common substrate (buffers) that can only hold one kind of thing (see the next point)
  • everything displays and manipulates a common data format (propertized text)
  • everything uses a common language (Emacs Lisp) so that functions, symbols, variables, macros, libraries, packages, etc can interoperate
  • everything uses a common set of flexible, but not too flexible, data types (hashes, vectors, lists, lambdas, symbols, keywords, propertized text, buffers), allowing further interoperation and ensuring that it is easy for different programs to adapt the data structures produced by others
  • everything has a common concept of the world around it (through the process, file management, network access, database access, and OS interface primitives)
  • everything runs in a common environment (shared memory, shared runtime, etc)

This choice – the choice of a set of primitives that is extremely flexible and powerful, but not infinitely flexible – is I think the core of what makes Emacs so incredible. It means that package or piece of code or data structure is reusable in new situations, because each new situation is never completely beyond their domain of applicability, and everything is maximally composable, because everything speaks a common language in a common context, so it can all interoperate seamlessly.

If any one of these things were too flexible – for instance, if pixel buffers were used instead of buffers of text, or different languages could be used, or the context was broken up – then all that composability, interoperability, and reusability would begin to evaporate.

Think about it this way: an buffer containing a stream of pixels is much more versatile than a buffer containing only propertized text which can have some static images overlaid on top of it. It can display more things, interact in a larger number of ways, etc. However, thanks to that infinite flexibility, there's no common way for different programs to understand that pixel buffer. You can copy it, flip it, crop it, rotate it, display it, and write to it, but you can't inspect it to pull out data or interface elements or semantics, because, well, it could be anything. There could be text in it, but it's just rendered down into individual pixels now, no semantic meaning preserved, and it could be drawn in any one of a nearly infinite number of ways, so good luck trying to reverse-engineer it. There could be images in it, buttons, menu items, but you'd have no way of knowing. Pixel buffers are a black box – once something's written to them, there's no getting it out again.

Meanwhile, something less infinitely flexible, like plain (English, Unicode) text, has more recoverable intrinsic semantics encoded into it: we have characters (letters, symbols, control characters) which are assembled into words and lines, arranged in a relatively predictable way, and probably human-readable. That's why text is so universal. However, even that is too flexible: given a raw stream of text, can you really figure out what the hell's going on? Not really. There certainly could be highly structured semantic data encoded into that text, but you have no guarantee it's in any kind of format you'd recognize, and even if it was, how would you determine that format, when you can't guarantee any kind of common out-of-band means of communication with the code you're talking to?

This is why being able to annotate text with a commonly understood set of data structures is so important: that way, you can attach semantic information, in a format you know other programs (within the world of the common set of primitives, at least) will be able to read and understand. This could be done by centering on an agreed-upon textual header or footer to append this metadata in-band, but in-band annotation comes with its own problems, and this annotation comes with the added hurdle of parsing and serializing it constantly. Providing out of band annotations in native data structures that can be human-readably serialized if necessary, but do not need to be, is superior, in my opinion.

Additionally, it is important that the information primarily visible to humans is still just plain text, and all of the most important content and data is still also just encoded in plain text – that keeps the flexibility and universality of text, the ability to take it elsewhere and to interpret it in a way that ignores the semantics of whatever produced it without having to do extensive and annoying conversions between highly structured data. We just need to be able to, again, imbue that text with meaning.

I also think this homogeneity is extremely important for human reasons, as well: a computing environment with a totally unified context, set of assumptions, data structures, libraries, and language – especially one designed for maximum composability and flexibility – is better for the human beings that use it, because it means that the knowledge we build customizing and building for ourselves can directly, smoothly scale up to the knowledge we need to build things for others or easily being able to understand and contribute to anything anyone else in the community is doing, which can then smoothly scale up to contributing to (most of) the core of the system – there's fewer points at which there's a fundamental break in paradigm, set of concepts, or programming language, where we have to use our limited human patience and time to learn something alien to us.

Unique user interface paradigm

Another really neat thing about Emacs is the unique user interface paradigm. There are a few aspects to this.

The first few unique aspects, which other users than me have also pointed out, stem naturally from the unity of internals and user interactions:

  1. Literally everything that can be done in the editor, or that the editor can do, is a command that the user can fuzzy search through and run. This is similar to the command palettes of things like Sublime Text, Visual Studio Code, and Obsidian, but infinitely more powerful, because it isn't arbitrarily limited to a set of commands that someone decided should be there – all the functionality of the editor is available.
  2. Everything that the user can do in the interface can be scripted – and not just through recording keypresses and dumbly replaying them, hoping that it works out right, but through the full power of Emacs Lisp to inspect editor state and conditionally do things based on that. This is because all user actions are just code and can in fact be easily recorded and exported as code to build on and add conditionality to.
  3. Every keybinding can be remapped, and new keybinding maps can be swapped in and out or overlaid on top of each other (with sparse keymaps).

There's something even more interesting than all that, in my opinion, though, represented by Magit, Transient menus, and applications like Dired and Proced. In this user interface paradigm, you generally have a consistent, pinned, stateful view of whatever content you're exploring or editing, a main "window" if you will, and then the ability to manipulate, modify, and navigate through it using single letter commands, without even needing modifier keys a lot of the time. These single letter commands can prompt you for input, in fact be trees of related commands, and can, with Transient, even take flags and toggle switches and secondary commands and input.

This means that you can get the higher interaction information throughput of the keyboard (as opposed to the mouse) for communicating what you want to do to the computer, while actually even increasing that throughput relative to something like the shell or key modifier-based interactions, since every command is just a single key press or a few key presses and nothing more, while getting discoverability and live, changing, stateful view of the state of whatever you're exploring or manipulating, like a GUI program, instead of having to follow the "do thing blindly, look at thing, repeat" loop of the shell, while getting the far greater expressiveness of Emacs key commands versus the flat key command hierarchy of GUI programs so that you can express more things with the keyboard.

To be fair, this user interface paradigm does show up in older Windows programs, I think, where you press Alt+<mnemonic letter> to navigate through menus to select commands you want to run on the content in the main window, but in my opinion not in as well-developed of a form.

A fascinating text editor

Although the rest of the "Emacs is an OS" joke claims that it lacks a good text editor, I think there is certainly an argument to be made that Emacs is a great text editor. It comes with a very rich set of concepts for editing text that puts any word processor and most text editors to shame – characters, words, lines, yes, but also sentences, paragraphs, sections (which you can structurally edit) via outline-minor-mode, and trees of expressions – all placed into a coherent keybinding schema that makes it relatively easy to memorize/remember/guess them, and very easy to look them up. It then also provides a powerful set of operations on each of these concepts, from transposition, to selection, to navigation forward and backward and up and down across them, to jumping to them, as well as an panoply of small utility functionality like zapping to characters, collapsing or expanding white space, and so on.

Yes, each Emacs command tends to encode a specific text object to operate on as well as the action you want to perform on that number of text objects into a single command, so by default you have to memorize MxN (text objects times actions) commands to use Emacs effectively, whereas in modal editors like Vim have an advantage in composability and learnability, because they have a text editing grammar you can use to compose a single mnemonic for the text object you want to effect plus a single mnemonic for what you want to do (plus numbers and adjectives just like Emacs) so you only have M+N commands. Concretely, for instance, if you want commands for deleting words, characters, sentences, and s-expressions, and then also for selecting, moving by, and copying them, in a modal editor there’d be a verb for deleting, selecting, moving past, and copying, and then objects for referring to words, characters, sentences, etcetera, and you can compose just those 8 commands/key bindings into all the actual commands you need, whereas in Emacs you'd have to memorize 16 separate commands. And in a modal editor, every new text object you learn or create automatically gets the benefit of all existing verbs, and vice versa, meaning extending the text editor’s range of concepts or learning new existing ones for yourself has a geometric instead of linear payoff rate, whereas in Emacs you have to define a whole new suite of commands for each thing.

However, this can be worked around: the core insight needed is that Emacs actually does provide all the components necessary out of the box to construct a text editing grammar: selections and region actions! Think about it: the way a text editing grammar works is that instead of specifying a single command that bakes in both the region and the command to be run on the region, you specify what you want to act on and how to act on it separately. Vim does action first and then text object, but that's not inherent to the idea of a text editing grammar – we could also describe what we want the other way around, object and then verb. And hey whaddya know, that’s how regions and their associated commands in Emacs work already: you highlight what you want to act on, and then say what you want to do to it, and Emacs already provides a command to select any text object it knows of, and a command that lets you perform any action it knows how to do on a region instead of a set text object (even supporting specifying multiple arbitrary objects through the use of the mark ring).

Of course, this wouldn’t work if the commands you have to select things were too primitive, like in most non-modal editors, because then the specification of what would be too low level, instead of specifying text objects you’d always be specifying either words or characters or, if you’re lucky, lines. However, Emacs does in fact offer a rich set of text objects baked into commands for selecting or moving by them (as well as doing other things to them, but those aren’t relevant right now).

All you need to apply this insight is to use it as a sort of mental jujutsu practice in existing Emacs – focus on learning and using the text object selection commands and the region action commands, or use the set-mark command and text object movement commands in conjunction with the aforementioned region action commands, if you wanted to also unify movement and text object selection (a good idea). Wax on, wax off! However, this could be made significantly easier, which is why I'm working on a minor mode to do this. It's working out pretty well – I'm daily driving it currently – the only issue is hammering out exactly all the places where such automatic selection after movement of the point by a command isn't wanted.

Moreover, even if it's pure editing language isn't quite as strong as Vim's, Emacs has incredibly powerful large scale text manipulation features like query-replace-*, which is like search and replace, but with many more features than most word processors and text editors could ever hope to achieve, or occur-mode which allows you to collect every occurence of a pattern across a buffer or set of buffers into a new buffer and do manual edits on that buffer, then have the changes written to each respective buffer, or grep which will generate find and grep commands for you, run them, and give you a hyperlinked buffer of the results, and allow you to edit that buffer and write the resulting changes out to all the relevant files transparently. And these are just a few of the powerful capabilities Emacs has!

I also think the fact that Emacs has structural editing commands, in the form of forward-sexp (move forward by expression), backward-sexp (move backward to/by expression), forward-list (move forward to/by an expression group), backward-list (move backward to/by an expression group), down-list (move into an expression group, so that forward- and backward-sexp move by its children instead of siblings), backward-up-list (the reverse of down-list), baked into its core, is incredibly important.

In Vim, despite its powerful and composable editing grammar, you still have to think in terms of a list of lists of literal characters. Therefore, you have to find your way to the next space, comma, or word to jump by sibling expressions; you have to navigate manually by lines and characters to go up or down expression trees; and the commands for doing that all have the specific syntactic elements of the language you're using (and even the specific layout of the piece of code you're editing) baked in. This means that there's a high cognitive overhead to editing in Vim: you have to consider specific syntax and positioning within the text, and the commands will be subtly but importantly different for every language you're working in. Hell, the closest thing Vim has to a high level structural command is merely to jump to matching braces with %.

Meanwhile, Emacs views your code natively as a hierarchical tree of expressions, with the syntax it uses to determine where identifiers and expressions begin and end not needing to be provided by you as you edit, but provided using syntax tables provided by the author of the language's major mode. This means that you can cut, copy, paste, navigate up and down and across, and transpose expressions without ever having to worry about the specific way the code is laid out or formatted, or even what the precise syntax is, in many cases. This is just infinitely more powerful and advanced than Vim, in my opinion. It hits the perfect sweet spot between true structural editing – which can be overly restrictive – and unstructured text editing like in Vim, where rather than limiting the transformations you can perform, is just an aid to your navigation and editing of code that you can use in addition. In essence, in my opinion, Emacs was ready for the Tree Sitter structural-editing-as-plain-text revolution 40 years before Tree Sitter was even a twinkle in its creator's eye!

More than just being a superior foundational set of concepts for text editing, requiring much less cognitive burden and allowing for more advanced manipulations, the fact that these structural navigation and editing commands are part of Emacs's core encourages, even requires, that authors of major modes for any language give some consideration to trying to provide meaningful and useful functionality for these motion commands. And luckily, thanks to the syntax table abstraction, the way in which Emacs figures out the meaning and structure of your file is abstracted from its commands. Its structural editing commands can gracefully degrade from Lisp to Tree-Sitter to Regular Expressions to simple character-based syntax tables to parenthesis/braces/brackets/etc, depending on what the major mode you're operating in supports, but no matter what the commands remain generally useful and applicable. This means that unlike in NeoVim, where you have to operate with a completely different set of text objects and motions when you have Tree Sitter available compared to when you don't – and the Tree Sitter objects and motions don't integrate will into the grammar – in Emacs you can confidently integrate its structural editing commands into your text editing vocabulary, safe in the knowledge that they will at least be somewhat useful everywhere you go.

Moreover, the structural editing commands provided in Emacs's core are all the building blocks you need to built proper structural editing on top, including things like soft deletion, slurping, barfing, splicing, convoluting, raising, concatenating, and more, all without ever having to rely on any language specific code – since all of that semantic and syntactic logic is already encoded in the syntax tables of existing language modes, which they need to do anyway to make font-locking and built-in Emacs movement work – or even rely on any specific parsing strategy.

Another interesting aspect of the Emacs text editing paradigm is that it focuses on being intelligent and automatic, instead of forcing the user to do all the thinking. While Vim (and similar text editor grammars) are excellent because they let you precisely specify exactly the text manipulation you want to do up front, in the editing command language, that's also their major downside: there is a high cognitive overhead to having to think about exactly what you want to do in a given situation, figure out how to express it, and then formulate it into terms Vim can understand. It also simply requires knowing a lot of individual things. Less things than the MxN explosion of first-order Emacs commands, certainly, but here's Emacs's secret weapon: it has second-order commands.

These are called DWIM (do what I mean) commands, and rather then asking you to precisely specify what you want to do, they try to intelligently figure out which of the first order actions you actually want to run. So, while most of the time it does offer the tools to precisely specify what you want to do if you need them – for instance, what to select, what to uppercase, etc – often the only commands you really need to memorize and keep in your pocket are the DWIM commands, which will figure out what you want to do on the fly.

Another example of this philosophy comes from an external package, expand-region: instead of specifying the text object or pair of delimiters you want to expand around, and whether you want to expand around or inside that text object, up front, like in Vim, and then having to replay the entire command over again if you get it wrong or want to adjust, expand region intelligently cycles through expanding your selection to encompass all of the relevant text objects at point, in order of size. This means that to get what you want, all you need to do is keep pressing the same key until you get what you need. It's the classic difference between recognition and recall.

A long history, yet still keeping up with the modern world

Emacs, in some form or another, has been around since, 1972. fifty two years ago. That's fifty-two years that this thing has been developed. Even GNU Emacs, which came into existence much later, has been developed for exactly 40 years at the time of this writing.

FORTY YEARS.

This age means a lot of things. Some are bad:

  • there is a lot of technical debt that holds it back (single-threaded core, ancient garbage collector, slow and old fashioned Lisp dialect, no promises or async/await for handling asynchronous programming in the standard library, no concept of using a pool Emacs processes as workers, therefore insufficient adoption of asynchronous programming into the package ecosystem, no ability to embed moving pictures or images that's cross platform, difficulty with drawing rounded boxes around text)
  • it uses terminology and keybindings that are unfamiliar to the modern world (kill for cut, kill-ring-save for copy, yank for paste, meta instead of alt, etc)
  • there's a lot of accumulated feature cruft – two entirely separate mail readers and two terminal emulators? – that can be confusing to people

But others are good:

  • it has time to grow and mature, each feature being fully realized and developed
  • it has had time to become incredibly feature rich
  • an absolute wealth of documentation, tutorials, accounts of experiences, and meta-level discussions of its philosophy and benefits have had time to grow and stockpile
  • the community has had time to build an amazing and incredibly complete ecosystem of packages, often with multiple well-implemented, feature-rich, documented, and robust ways to do common things people want like reading mail, doing structural editing, taking notes, writing LaTeX books and papers, managing citations, as well as complete and almost unbeatable environments for programming languages like OCaml, Haskell, Common Lisp, Clojure and so on, as well as an incredibly long tail of generally simple and robust, or at least mostly workable, packages for anything else you could ever possible want thanks to the "scratch my own itch" model of free software development
  • the maintainers have been able to achieve a perfect balance, over time, between moving Emacs forward at a reasonable pace (and in my opinion they've done an incredible job of that in the last several versions), and maintaining stability and backwards compatibility in this tool that so many people have built their entire computational lives around; moreover, these maintainers have been able to develop a sort of institutional wisdom, passing down a clear-eyed understanding of what makes Emacs special and how to preserve that
  • if Emacs has already managed to last for forty years, it is likely to continue to exist for a very long time.

Rich plain text

Thanks to the ability to propertize text, and its advanced display system, while Emacs is text-based, it is unapologetically rich-text based, in a way where it can not only display that rich text, but actually edit and manipulate it on the same level as plain text, and convert back and forth, in-editor, by the user if need be, in a way other editors can only dream of. Other editors may be able to make plain text look richer, but you won't be able to meaningfully inspect, edit, and manipulate that rich text style, or sometimes even the text when richly displayed like that; others may be able to display rich chrome around plain text, but not display the plain text itself richly. Emacs can do all these things and more. Here's an incomplete list of the things display properties can do with plain text in Emacs:

  • Pixel- or character-wise left, right, and center align
  • Set a constant character- or pixel-wise height or width to crop to
  • Overlay an image, PDF, or SVG (or even a GTK+ widget)
  • Change the font size, face, color, and style (even within a single line)
  • Add hover text
  • Change the mouse cursor when it's over the text
  • Add a box, underline, or overline
  • Add box margin
  • Change the fringe next to that text
  • Raise or lower the text from baseline
  • Turn it into a link or a button
  • Change the background color
  • Turn it into a text box
  • Replace the text

On the subject of the Emacs display system, by the way, how does it work?

Well, for instance, I recently decided to switch dashboard packages to something faster, but it didn't implement image banners by default — so what did I do? The new dashboard package obviously lets you pass in structured data (lists and plists) to decide the structure of your dashboard, where there are strings that it uses to print out the menu items, but how could I extend this to images? Well, it turned out to be extremely simple: I just had to make the headline menu item a string with metadata attached to tell Emacs's display system to display an image on top of it. That's right — images are first-class objects in Emacs, and all you need to do to display them is attach them as metadata to a string. Note, this doesn't produce a new kind of string object that some parts of the language or editor know how to interpret and some don't, and when it is output, you don't lose the text or the metadata — it's just a regular string, imbued directly with rich multimedia metadata, that can be treated like a regular string by your code, while also showing up as an image in the display, from which the original text or metadata can be derived. Here's the first part of the code:

(concat
          "    "
          (propertize "QUAKE EMACS" 'display (create-image "~/.emacs.d/banner-quake.png"))
          "\n"
          "\n"
          (propertize  (format "Started in %s\n" (emacs-init-time)) 'face '(:inherit 'font-lock-comment-face)) ...)

That's right, you can just use the regular string concat function to concatenate together images and text, because images are just text with metadata attached! Here is how it looks:

./static/emacs1.png

And this is how the entire UI works: it's all just structuered data where the leaves are texted annotated with metadata to control its display and interactivity. This metadata principle is how you make text taller, or turn text into buttons and attach handlers to those buttons or draw images or different fonts or icons or display arbitrarily-shaped and sized GUI elements using SVG or embed GTK+ widgets — the entire interface is just structured lists of text with metadata attached that turns that text into rich UI elements in a predictable and deterministic way; and you can get the original text and associated metadata that was used to construct a part of the user interface back out of it at any time, just from the UI itself. Combined with the fact that you can attach arbitrary metadata to that structure text — and in fact that Emacs programs often do so — and also tell the display engine how to interpret new types of metadata, and you have an extremely rich way of building introspectable interfaces! And in the end, you can just use the regular string or list processing functions on all of it, buttons, images, various sizes and fonts of text and all, because images are just text with metadata attached.

This system is even more dynamic and introspectable than HTML is — especially since you can attach arbitrary metadata, so it is common practice to attach the relevant internal UI state as metadata to each UI element you push out to the screen, meaning that any program can reconstruct the state that generated any UI that it is looking at. Not the deep global application state of course, but the stuff conceptually "right behind" the UI anyway. The power of having the end-product user interface your program generates not only be generated from native language data structures and constructs, but also be derivable from that generated UI, so that the native language nature of it all is bidirectional, is incredibly useful, almost mind-bogglingly so to me. HTML/the DOM/JS gets close to this — that's a huge part of why the web stack is so fundamentally powerful and popular — but in my opinion it is a bloated, poorly designed in many respects, version of this, not intended for user interfaces initially; in my opinion, it is also less powerful and interesting, partly because HTML and the DOM aren't very native to JS in actual fact, and partly because with HTML, determining how to interact with and display content comes prior to the content, instead of being attached after.

Yet, the best part is, it isn't just a dumb overlay, or an image broken into little character-size pieces, like in terminals that can "display images": this is a full single image (jpg, png, svg, what have you), rendered as a single image, with the text around it flowed to allow it to fit properly, just like on the DOM (although the reflow algorithm is a lot more predictible imo: the image is a single "logical line", the line is just bigger). The best part is — if you're in an emacs lisp REPL, like eshell, where the data structures returned from the expressions you are evaluated are pretty printed…

./static/emacs2.png

yup that's right, native support for return values that are images! You can even wrap this in a command and use it to cat images. Here's how you'd dynamically extend the built in Emacs Lisp shell's cat command to be able to cat images and Emacs buffers as well as files in 13 lines of code:

(advice-add 'eshell/cat
            :around (lambda (oldfun &rest args)
                        (mapconcat (lambda (arg)
                                       (cond
                                        ((bufferp arg) (concat
                                                        (with-current-buffer arg (buffer-string))
                                                        "\n"))
                                        ((string-match-p (image-file-name-regexp) arg)
                                         (concat
                                          (propertize "image" 'display (create-image arg))
                                          "\n"))
                                        (t (apply oldfun args))))
                                   args)))

And here's me creating and returning a button object, which is then pretty-printed as a button by the REPL (this isn't me inserting the button, I'm just returning the button object and it's displaying it) with a callback that runs when clicked:

./static/emacs3.png

(You could give the buttons outlines and depth and stuff, but I don't bother.)

In a lot of ways, it's like having the stuff iPython Notebook gives you, but as a more powerful, generalized system you can build whole interfaces with instead of it just being a special-case of a REPL that can render return values — it's a whole rendering engine that works based on the metadata of objects passed into it, and you can combine them or add your own.

And all of this is simply how the Lisp Machines worked to produce their entire display.

The culture

One of the things I love most about Emacs is its community culture. Over the forty years Emacs has been around, the Emacs community has had time to build a rich, unique culture: a mature set of norms and expectations, a deep appreciation for and understanding of the nuances and pros and cons of Emacs and everything that it can do, and an understanding that it may not be for everyone (they've long since abandoned the editor wars).

The two things I especially love about this culture are:

  1. Its deep and pervasive commitment, at least for many packages, to complete documentation of their code, and long, comprehensive, robust, well-written, and erudite user and reference manuals for major packages that have a lot of complex functionality. One of my favorite things to do is sit down and read through an Emacs package README, they're so often quite excellent.
  2. That Emacs's usage is not dominated by software engineering grognards (like Vim) or software development hipsters (like NeoVim) or even software engineers in general (like Visual Studio Code). Instead, its community is almost equally composed of scientists, researchers, data scientists, teachers, and people from the humanities such as philosophers, very many writers and novelists, journalists, social scientists, solo RPG enthusiasts and even musicians. And it is so flexible and responsive to their different needs, preferences, and workflows, that such people can really bring what they use Emacs for to the community, and integrate what they do with Emacs, instead of just silently happening to use it on the sidelines. This leads to a richer, more vibrant, more interesting, and often even more humane culture – and, as someone who approaches software engineering from a more humanities-oriented lens, a culture that is more stimulating and fun for me.

A sort of corollary to these two points, just in general, is that the Emacs community is intensely literate, erudite, well-spoken, and academic, from what I've seen at least. This is just… really pleasing to me. Others may find it off-putting or dry, but for me it just means that almost everything it produces will be a pleasure to read or listen to, engaging, full of interesting ideas and information, and written in an educated, complex style that is enjoyable to read. I don't know if Emacs's heritage in the MIT AI Lab, which was composed of erudite, literate academics and professors, and which spawned off quite literary endeavors such as Infocom, might be influencing this, but it certainly feels like it.

Programmable, not (just) customizable

Although Emacs Lisp has a lot of capabilities for defining well categorized, grouped, documented, typed sliders and customization variables for packages, and a very advanced and relatively user friendly GUI for manipulating those customizations variables, the power of Emacs is that the predominant way you customize for your needs and workflow it is by programming it. This means that your customizations can be intelligent, conditional, and far more versitile, and it also means that there's an easy on-ramp from simply customizing Emacs to really using it as a full-scale automation tool and computing environment.

Moreover, a tool that is merely customizable has to think of every possible thing someone might want out of it in advance and explicitly program for all of those things. This is both an impossible, Sisyphean task that will always leave someone dissatisfied, and also creates a combinatorial explosion of features and settings to account for and test together, which will inescapably lead to massive, sprawling, buggy codebases. Meanwhile, a programmable tool takes the wise way out: it can focus on making core functionality powerful, composable, accessible, flexible, and versitile, but doesn't have to worry about every behavior or use someone might want out of it: as long as it makes sure to maintain backwards compatability, provide documentation, and make sure modifications can compose robustly, it can easily grow and change to fit everyone while being able to just focus on its core functionality and keeping everything straight. This video says it very eloquently.

More importantly, this programmability, where absolutely anyone can come along and replace, modify, or expand on even the deepest and most fundamental parts of the editor, is part of what's guaranteed Emacs's longevity. Whenever another editor comes out with important functionality, whether that be autocomplete popups, snippets, and fuzzy vertical completion UIs for commands and filenames back in the distant past, to embedded terminals, to side bar file browsers, to language servers, to tree sitter, to multiple cursors, to project management, Emacs was able to (more or less handily) absorb this functionality. Sometimes it wasn't as good as the original, sometimes it was better, but the point is that if there was some piece of functionality that you wanted, you didn't have to switch away from Emacs if you didn't want to, and the rest of the Emacs computing environment would make up for whatever minor deficiencies there were in its implementation of these features. Basically, the more programmable and malleable your editor is, the less likely it will be to suddenly lose popularity and fall into the dustbin of history as people go chasing the next new feature.