Emacs should *never* become multi-threaded

Emacs should never become multi-threaded   emacs

There's a lot of discussion in Emacs community around how Emacs should become multi-threaded. While I think a lot of these ideas are interesting and could work, I think they're ultimately misguided. Most of the fundamental appeal of Emacs is that, at its core, it is a big, huge ball of mud – and by mud, I mean global mutable state. The fact that it's just this huge ball of state that anything can access and modify at any time, from anywhere, and immediately see the changes live, without hindering the access and modification of that state by any other part of the program, is what allows Emacs to be what it is in the first place. But that kind of global, shared mutable state is just fundamentally incompatible with multithreading, since it leads to race conditions, instability, memory corruption, conflicts, and in general just extremely difficult to track down and resolve bugs. And any solution to that problem would fundamentally chip away at what makes Emacs unique:

  • Implementing zero-shared-memory message-passing, like you can find in languages like Erlang: essentially reduces being able to modify the global state, or the state of other programs, to a limited, rigid, non-transparent API – at that point, you might as well move packages out into plugins and extensions communicating with each other and the editor over RPC like NeoVim does.
  • Implementing exclusive memory access systems like locks and mutexes on buffers and their associated local variables: drastically reduces the usefulness of Emacs's unique (compared to Vim and NeoVim at least) ability to run multiple modes (one major mode plus N minor modes, or even multiple major modes on different regions) on a single buffer, imbuing all kinds of different overlapping meaning and functionality into buffers in a way few other editors can match. Additionally it doesn't solve the problem of dealing with non-buffer-local variables, and introduces the possibility of deadlocks, which are an almost as severe problem as race conditions!

Instead, I think Emacs should lean into what it already does well: shelling out to external processes when parallel execution (for example, super heavy long running computations) is needed. This is already an extremely common pattern in Emacs programming, and one that I think should be enhanced to make as easy and ergonomic as possible, through two methods:<<Multiprocessing Suggestions>>

  1. Emacs Lisp's standard library already has extremely powerful primitives for this kind of functionality, but they're ultimately pretty unwieldy and verbose, require a lot of boilerplate and esoteric knowledge about process-management terminology that may seem natural to almost any computer nerd worth their salt but will be alien to the humanities end of the Emacs user base, and just aren't good enough yet. Thus I think integrating things like emacs-aio, which allows you to do async (based on Emacs generators), mutithreaded, and multiprocess programming using a common library interface that's extremely easy and generally works and looks like synchronous code that we're all familiar with, and integrates deeply with Emacs's command loop, is probably the way to go.
  2. Allowing Emacs Lisp programmers to shell out to external Emacs processes to do work more easily by including something like the emacs-async package – but that can allow Emacs instances to communicate data structures (and even closures and callbacks) using raw bytecode instead of needing to serialize and deserialize S-expressions from buffers – in the Emacs Lisp standard library. This gains you the asynchronous programming benefits of multiprocessing and shelling out, without having to lose the unified language, set of data structures, interoperability, and consistency, of the Emacs computing environment.
  3. Creating a pool of Emacs worker processes that can all pull jobs from a central queue (and maybe do work-stealing between each other), for the purposes of making point 2 more efficient.
  4. Making the core of Emacs (the display code and Lisp interpreter) more multithreaded. This shouldn't effect what "user space" Emacs is like at all, but could prevent things like long lines or garbage collection from blocking everything else.