Do we really need macros much?
I've kind of fallen out of love with macros to a certain degree.
Now that we don't just have snippets, but also AI, to help deal with boilerplate and refactoring, to a certain degree, yeah — but there's more to it than that.
Ultimately, it's because the vast majority — like 99% — of things you do with macros in Lisp, are just basic textual compression, basically. You get rid of boilerplate, but the resulting DSL is completely isomorphic to the structures and control flow of the underlying code.
In theory, you technically could maybe add brand new ways of doing, for example, control flow, using macros, but in reality, to properly implement meaningful changes in the language's paradigm or capabilities, you need runtime or compiler support. Trying to do it all through macros leads to you having to redefine or wrap so many fundamental pieces of the language that you end up writing an entirely new language that just happens to compile to Lisp; and the bigger and more fundamental the change is, the more the rest of the language has to be wrapped and abstracted to interface with it; so it can become this eternally growing slime mold of macro abstraction, which ends up being about as much work as implementing a brand new language (since you're having to reimplement nearly every construct, and implement a new runtime, essentially, that compiles at macro compilation time to a runtime that doesn't even have the same features, so your Lisp essentially becomes bytecode), for more brittle and leaky abstractions over the core of Lisp. This is how you end up with something like Coalton. Or you end with a sort of sub-language DSL that can't fully interact with the host language without all kinds of interoperation boilerplate everywhere. This even happens with something like Clojure's core/async; to the degree that it doesn't become macro slime mold, it's because it becomes colored function slime mold: you have to deal with the interoperation/integration barrier between async/await macros and regular code yourself; if you wanted CSP in Clojure that works as well as it does in Go, with automatic halting of concurrent lines of execution and so on, it would turn into a Coalton-alike.
Macros are just code arbitrarily reading and constructing new code as an untyped, fungible tree abstraction: powerful, but also brittle and difficult to debug or inspect. If you're trying to do a meaningfully complex and nuanced transformation on code that literally could look like anything — god help you! Think about nested and recursive macros sometime, or scoping issues, or phased compilation like Racket has, or trying to get good debuggability and error messages out of a macro that someone else wrote, or write good ones for your macros. And that's not even to mention how macros utterly destroy IDE support. Where's go to definition, find references, extract to function, extract to variable, hell even rename, in the presence of a custom macro DSL? The "productivity" gained by writing fewer lines of code is often lost the moment you need to debug a macro-expanded stack trace, or you lose IDE support.
And it's worse if you try to integrate with the host language: for instance, all the attempts to implement parametric polymorphism for Lisp types or classes, or monomorphization or static inline dispatch or JIT compilation for Lisp method dispatch: they technically work, but they're a difficult to maintain hack, a leaky abstraction, that just can't reach the level of complete implementations that handle all the features and edge cases, and the integration with the host language, of a language that had them from the start. Ultimately, all macros will ever be are an isomorphic transformation to existing underlying control flow and concepts, or a leaky, brittle hack that causes more headaches than it saves you.
And this is important to reiterate: the uniquely useful thing that macros can do is so frowned upon for an average project to do, and there's generally a strong push to standardize these things so that everyone recognizes, understand, and uses these things, so that it usually ends up being language authors themselves that provide macros. Even Lispers, at this point, know that you really should be keeping macros to a minimum; preferably using none at all. You even see this with Clojure, where the big Clojure type systems and async IO implementations and so on are generally provided in the standard library itself. At which point it's not clear to me why you wouldn't want to just implement it in the language in the first place, except for purity reasons.
So if macros can't add fundamentally new features to the language that you might need, what do they do? Well, think about the fact that most Schemes only have — or at least heavily encourage, and mostly only use — syntax-case style macros, where all they can do is some basic pattern matching and template expansion. It's clear that what's left to macros is just simple text compression, shallow and mostly standalone DSLss that don't need to define totally new semantics or control flow, nor interoperate with the host language much. IE, regex builders, SQL query builders, configuration formats, HTML builders, etc. But even /Go/ can do a decent DSL impression! It's not really unique to macros.
For instance, Clojure has a macro for threading:
(->> data
(filter pred)
(map f)
(reduce g))
But this can, with higher debuggability and the same conceptual affordances in terms of data flow, be represented with variables:
filtered := filter(data, pred)
mapped := map(filtered, f)
result := reduce(mapped, g)
Or, with similar concision, but without a macro abstraction, using interfaces or universal function call syntax:
result := pipeline(data).
Filter(pred).
Map(f).
Reduce(g)
And it's questionable whether you want regex and SQL builders, instead of just learning to use regex and SQL, which are generalizable skills and give you more direct access to what you're doing.
And think about the fact that, if you're trying to model a domain, in most languages that aren't outrageously underpowered, I've never really felt the need for macros over just higher order functions, structs, interfaces, generics, enums, decorators, methods (with things like the builder pattern) and so on to express concepts I want to express from my domain model directly. Because those are the underlying things macros would use anyway. Even in a language like Clojure, with excellent and very easy to use macro facilities, I've never really used them and to the degree I have, they're from libraries where the The same thing could have been implemented relatively as easily, with only marginally more boilerplate — like one or two extra tokens of boilerplate — without them. In fact, in most cases where I've written any substantial amount of Lisp, Scheme, or Clojure, whenever I've created a macro, I later realized I didn't need it, and refactored it into a regular function.
I also often feel like, most of the time, macros are really made necessary only by an insufficiently concise or expressive host language. You use macros because a language is too verbose, or because it doesn't have a feature you need. But in the latter case, as I've discussed, you don't end up getting a very good impression of that feature. And in the former case, it's not the god-like productivity boon Lispers like to claim it is; at best it's a nice-to-have, and would itself be elminated by a better language. Languages that have good syntactic sugar for lambdas or other forms of delayed evaluation to pass into functions, or quasiquoting, or universal function call syntax, or operator overloading (or arbitrarily-named and concise methods), like Smalltalk, Ruby, Elixir, etc, don't ever feel the need for macros, even for text compression purposes.
So, really, to me, it feels like macros are an unsatisfactory solution to problems in the language that shouldn't be there to begin with, and we don't/shouldn't need to build our own language features anymore because we can use languages that already have the features we need, implemented properly.
The rest of Lisp's features are similar: extremely cool in theory, but, even after getting a chance to use them, lackluster and generally not useful in practice. Multiple dispatch would be nice, to avoid the visitor pattern, but structurally-typed interfaces (like in Go) satisfy 90% of that need for me; image-based development would also be nice, but has its own pitfalls (the state of your program and the state of the file, in terms of what code is actually running, getting out of sync, which can be a horrifying headache) and in general is easily replaced with simple hot reloading (which is always in sync!) and standalone static binaries to avoid dependency hell.