Emacs should shell out

Emacs should shell out   emacs

There are two approaches to adding tools and functionality to Emacs.

The first one is to take an existing program with IPC functionality or a command-line interface – usually most profitably the latter, for as much as I think the UNIX philosophy is incomplete and short-sighted, the idea of making sure that as much program functionality as possible can be called from the command line, and having everything communicate through text streams, ensures the composability and scriptability of programs far better than most other paradigms – and write an extensive library of Emacs Lisp functions and defcustom variables wrapping that functionality, providing a way to deeply script, control, and modify the behavior of that program, as well as a set of transient, hydra, or in-buffer user interfaces for that program.

The other is to completely reimplement the functionality of that program in Emacs Lisp from the ground up, using the powerful standard library and platform-agnostic language runtime provided by Emacs itself, providing a self-contained package that functions like a fully independent program installable through the magic of use-package or package-install.

Each of these approaches has benefits and drawbacks.

The first approach's main benefit is that it allows Emacs to take advantage of the already-existing powerful and featurful tools that exist outside of it, while still lending most of the benefits of the Emacs computing environment to those tools. Thus, Emacs can benefit from all of the functionality that those tools have already implemented, and all of the new functionality that they will implement in the future, essentially for free. The maintainers of the Emacs package also get to benefit from sharing the maintenence burden for that functionality with others outside the very small Emacs community, and users of the Emacs package get to benefit from using the same tool that a large community of non-Emacs users also use, such as greater interoperability, more documentation, and more access to help. The main drawback, of course, is that since the tool is not implemented from the ground up in Emacs Lisp, only the functionality and customizability that is exposed through its IPC or command-line API can be accessed through Emacs Lisp. Deeper core functionalities can't be changed, and new functionalities must be layered on top in terms of the actual capabilities of the program itself. In essence, it works like adding more core C code to Emacs, placing an extra boundary on what it can do. The secondary drawback is that having your Emacs computing environment rely on external tools can be a recipe for fragility when you take it with you from place to place, since the programs you've made it rely on may not be as platform independent as Emacs itself is.

The benefits and drawbacks of the second approach are essentially precisely the inverse of those of the first approach: while implementing a tool completely in Emacs Lisp means that it is completely hackable from bottom to top, and much more self contained in the cross-platform runtime Emacs provides, it means that the burden for feature implementation, improvements, and maintenence falls entirely on the package authors, and the burden for documentation and user help falls entirely on the much smaller user community, and interoperability with what other people are using is sacrificed. There's also another drawback to be considered: the fact that, while Emacs Lisp is certainly not the slowest language around – it's a lot faster than Python or Ruby, which are both used to build quite large applications themselves – it's still a lot slower than you might want for very large and complex user-facing applications, especially in a painfully single-threaded environment like Emacs.

As a community, there's no need to choose between these two strategies wholesale, of course; for any given thing you want to do, packages that embody both – such as Gnus (the second strategy) versus Notmuch or Mu4e (the first strategy) – will always exist to choose from, and that's a very good thing! From a user perspective, which of these strategies to choose will depend on one's personal preferences and assessment of uses and needs on a case by case basis.

However, while I don't ever want packages using the first strategy to go away – I use Gnus without anything like offlineimap myself – I do believe that new package authors should strongly prefer the first strategy over the second. While it does compromise the core values of Emacs, the entire history of GNU Emacs beginning with its own core being written in C is one of compromise, and until several issues at the core of Emacs are resolved, that compromise must continue in the name of making sure that Emacs remains pleasant to use as well as powerful.

In this case, compromise must be acceded to because of the fundamental truth that Emacs is a very small (but dedicated) community even within the niche community of text editors, so we simply don't have the resources or manpower to maintain things and establish interoperable standards on our own. To see the truth of this, just consider CEDET, which was an effort to expand Emacs with IDE-like capabilities including project management, language-aware but agnostic source code analysis for refactoring, jumping, and completion, and a lexer and parser generator for language-agnostic structural editing: it quickly died and became abandonware, because no other communities were using or contributing to it and no languages were interested in supporting it just to help the Emacs community out, so all the work fell exclusively on the Emacs community and it failed to boostrap enough interest. Compare that to the success and revitalization Emacs has seen in recent years thanks to the adoption in core of LSP and Tree-Sitter, standards providing the same functionality that were instituted and supported by other editors such as Visual Studio Code and NeoVim, and the greater success of smaller and more focused packages for single IDE features like Treemacs and Projectile. Adopting technologies and programs used by other communities ensures that your package will have a much longer, more reliable, more robust life, which reflects on Emacs's usefulness and longevity! Furthermore, the first strategy only enhances one of the unique features of Emacs, which is that it is essentially a new and different kind of shell, not just a hermetic computing environment (the most similar thing I know of outside of Emacs is Acme, although it suffers from UNIX philosophy nonsense), so it's not a pure loss.