The failure of tree-sitter for structural editing

I love text editing in Lisp. The clean beauty, efficiency, and confidence of the structural transformations you can do on S-expressions is just unbeaten.

I have long wished that tree-sitter would provide a way for structural editing of that caliber to come to other languages. However, more and more, I think that while tree-sitter can offer a some new extremely useful tools for text editing in other languages, it will never truly get to that level of power and polish. While I used to have hope for tree-sitter to provide a similar level of structural editing for other languages, at least in most editors I've just not found that to be the case. There seem really to be two ways to use tree-sitter to add structural editing to languages: one, to write custom queries for every language, in order to get Vim style syntax objects, and two, to try to directly move/select/manipulate all nodes in the concrete syntax tree as if they're the same, essentially trying to treat tree-sitter's CSTs like S-expressions.

The problem with the first approach is that you end up with really limited, often buggy or incomplete, language support, and structural editing that requires a lot more cognitive overhead: instead of navigating a tree fluidly, you're having to "think before you act," deciding ahead of time what the specific name, in this language, is for the part of the tree you want to manipulate. Additionally, this approach makes it much more difficult to do more high level, interesting transformations; even simple ones like slurp and barf become a bit problematic when you're dealing with such a typed tree, and more advanced ones like convolute? Forget about it.

The problem with the second approach is that, if you're trying to do generalized tree navigation, where you're not up-front naming the specific thing you're talking about, but instead navigating the concrete syntax tree as if it's S-expressions, you run into the problem the author of Combobulate and Mastering Emacs talks about: CSTs are actually really different from S-expressions in practice, because they don't map uniquely onto source code text; instead, they're something overlaid on top of the source code text, which is not one to one with it (in terms of CST nodes to text token), but many to one, because the CST is very granular. Which means that there's a lot of ambiguity in trying to understand where the user is in the tree, where they think they are, and where they intend to go.

There's also the fact that tree-sitter CSTs contain a lot of unnamed nodes (what I call "stop tokens"), where the delimiters for a node of a tree and its children are themselves children of that node, siblings with the actual siblings.

And to add insult to injury, most language syntaces just… don't really lend themselves to tree navigation and transformation very well. Conceptually, most of the structural editing transformations make sense in other languages, but — and this is something I'm probably going to have to come back and explain later if I can find the words for it — their syntax and constructions never seems to fit a tree quite right. I think it's the infix syntax: you'll try to traverse an expression, and end up traversing its operands and the operator, instead of the whole thing at once, because the language doesn't really represent expressions as trees instead of soups of tokens, for instance. The operations just never seem to function, especially motion, like you'd want them to for maximum efficiency; you end up spamming motion commands over and over to get through a muddy slurry of random tokens.

I actually tried to bring structural editing to a level equivalent to the S-exp commands in Emacs recently, but ran into all of the above problems. I recently moved to Zed, and while its implementation of structural editing and movement is better than mine, and pretty close to 1:1 with the commands available in Emacs (especially if they accept my PR), and also takes the second, language-agnostic, route, it's still not as intuitive and reliable as I'd like.