Writing1 August 202612 min read
Metaprogramming: An elegant weapon for a more civilized age
The thing about lightsabers, and the films rather skate over this, is that a Jedi has to build his own.
There is no quartermaster. You go to a cave on an unpleasant planet, find a crystal prepared to tolerate you, and assemble a device that will, if you have made a mistake, remove your own arm at the shoulder.[1] The weapon is elegant because the person holding it made it and knows which bit is likely to go wrong.
Most days we write programs. Occasionally we write the program that writes the program, at which point the job changes character entirely. The tool stops being something issued to you by a committee in another country and becomes something you shaped, which is thrilling for about a fortnight and then becomes a responsibility.
The oldest idea in the building
In 1936 Alan Turing described a machine, and the machine was essentially a very stupid clerk. He has an infinitely long ledger, a few rules, and no curiosity whatsoever. He reads a symbol, consults the rules, writes a symbol, shuffles one square along. He is not paid well and he does not complain.
Then Turing did the thing the rest of the century has been living inside. You could write, in the ledger, a description of a different clerk, and the first one, following his own rules with the same absence of imagination, would behave exactly like the second. The description is data. Being a machine is what happens when something reads the description.
Von Neumann put this into hardware, where instructions and numbers share one memory, in the manner of a village whose fire brigade and darts team are the same six men and nobody has ever found this confusing.[2]
You are compiled from source
Your genome contains the instructions for building the machinery that reads your genome. Worse, a lymphocyte cuts up the DNA it was issued at birth and reassembles it into a receptor gene that never existed in any ancestor. Something has come through the door that nobody has a form for, so the cell writes a new form out of the old one.
The prequels tried to explain the Force using biology and the internet has been sulking about it for twenty five years. Biology got there several billion years earlier and nobody wrote a single angry letter.
A macro is a function that keeps unusual hours
A macro is a function. That is most of it. It takes arguments, it has a body, it returns a value. Two things differ, and both are about timing.
It runs while the program is being built rather than while it runs. And it does not receive the value of its arguments. It receives the code you wrote, handed over as a data structure, still in one piece.
Suppose you write assert(total == expected).
A function called assert is a messenger who has been told the answer and has forgotten the question. It receives false. The names evaporated, the numbers evaporated, and the most it can tell you is that something, somewhere, was not true. Everyone reading this has met that message at two in the morning and said something unprintable to a screen.
A macro called assert receives the whole conversation.
you write: assert(total == expected)
a function receives: false
a macro receives: (== (var total) (var expected))
Holding the tree, it can reach in, pull out the names, and return new code that performs the comparison and, on failure, reports both names alongside both values. The compiler puts that code where your call used to be and carries on as though nothing has happened. By the time the program runs there is no macro anywhere in the building, only the code it left behind.
This is also the difference between a library and a language extension. A library hands you new verbs inside a grammar somebody else fixed years ago in a committee room. A macro lets you argue with the grammar, because it decides whether to evaluate its argument at all, which is a power an ordinary function does not have and cannot be lent.
What varies between languages is how the tree looks when it arrives and how much of it you may touch without a licence.
Yoda writes Lisp
Lisp is around seventy years old, which in this industry makes it a geological feature, and its detractors have been making the same joke about parentheses for most of that time.[3]
The obvious gag is the word order. The interesting part is that in Lisp your program is a list. Not stored in one, not compiled into one. It is a list, in memory, in the same form as your data, so a program can take another program apart using the functions it uses for shopping.
(defmacro aif (test then &optional else)
`(let ((it ,test))
(if it ,then ,else)))
(aif (find-user id)
(format nil "welcome, ~a" (name it))
"no such user")
Look at what became of it. The macro introduced a name the caller never declared, and the caller uses it as though it had always been there. This is anaphoric capture, it is deliberate, and it would get you removed from the premises in most other languages. Lisp permits it because Lisp has no opinion about what you ought to want.
Nine hundred years old, two feet tall, holds the sentence together back to front, still the most dangerous thing in the room.
Han Solo runs the C preprocessor
The preprocessor is not a bad man. He is a very good man at a job nobody should have given him.
He runs before the compiler has the faintest idea what is going on, he substitutes text for other text, and he is quick when the situation is deteriorating. Write #define MAX(a,b) ((a)>(b)?(a):(b)), call it as MAX(i++, j++), and one of your variables gets incremented twice. Somebody else will find out, on a Sunday, in a payments system.[4]
And yet here he is doing the thing he is rather good at:
#define ERRORS \
X(OK, "fine") \
X(TIMEOUT, "took too long") \
X(REFUSED, "declined by upstream")
typedef enum {
#define X(code, msg) ERR_##code,
ERRORS
#undef X
} error_t;
static const char *error_text[] = {
#define X(code, msg) msg,
ERRORS
#undef X
};
The list exists once. The enum and the message table are grown from it and cannot drift apart, because there is nothing left to drift. The trick is older than most of the people using it and needs no build step, no dependency and no meeting.
He has never heard of a type. He holds no view on scope. He will take your foot off while insisting he had right of way. He is also, now and then, precisely the right man for the job, which is the part people leave out of the eulogy.
Palpatine and the C++ templates
Nobody sets out to build an Empire. Empires are what you get when a series of entirely reasonable proposals are approved by people who are thinking about lunch.
Templates arrived as a modest suggestion concerning generic containers. Then in 1994 Erwin Unruh wrote a program that calculated prime numbers and delivered them as compiler error messages, demonstrating that the template system was, by accident, a complete programming language nobody had designed or voted for.
template<int N> struct Fib {
static constexpr int value = Fib<N-1>::value + Fib<N-2>::value;
};
template<> struct Fib<1> { static constexpr int value = 1; };
template<> struct Fib<0> { static constexpr int value = 0; };
static_assert(Fib<20>::value == 6765);
Read that again with the correct sort of horror. The recursion is the template, the base case is a specialisation, and the arithmetic happens inside the compiler in a language with no grammar book. Not one machine instruction is emitted for any of it.
By then the clone army was on order. Every instantiation produces another identical unit, requisitioned by nobody in particular and charged to your CI budget. The compiler labours through the night and the error messages arrive at breakfast, four hundred lines of them, patiently describing a type whose name will not fit on the monitor.
Mace Windu carries Rust
Rust looked at the preceding two chapters of history and did what any sensible civilisation does after a large fire. It wrote a building code.
macro_rules! hashmap {
($($k:expr => $v:expr),* $(,)?) => {{
let mut m = ::std::collections::HashMap::new();
$( m.insert($k, $v); )*
m
}};
}
let config = hashmap!{ "retries" => 3, "timeout" => 30 };
It matches on syntax rather than text, and the trailing $(,)? forgives a stray comma, because Rust is strict but not gratuitously cruel. The important part is invisible: that m inside the expansion cannot collide with a variable called m in your code. What Lisp allowed on purpose with it, Rust forbids on principle.
Anything more ambitious becomes a procedural macro, which means syn, quote, a crate of its own, and a form filled in before you may proceed to the next window.
Windu is precise, senior, does not smile, and will stop you at the door when the paperwork is wrong. He is also right almost every time, which is the most irritating part.
R2-D2 speaks Elixir, C-3PO does code generation
They travel together and solve the same problem in opposite ways, which makes them the most useful pair in the canon.
C-3PO translates from the outside. He is your schema driven code generator: you give him a specification, he produces fourteen thousand lines of impeccable, correct, entirely unread code, he mentions the odds, and everyone continues talking over the top of him.
R2 does not translate. He plugs into the socket and speaks the machine's own language from inside it.
The whole of Elixir is three element tuples: operation, metadata, arguments. Every macro is a function from tuples to tuples, running during the build, which is all anybody needs to construct the thing Elixir people construct roughly once a fortnight:
defmodule Router do
defmacro __using__(_opts) do
quote do
import Router
Module.register_attribute(__MODULE__, :routes, accumulate: true)
@before_compile Router
end
end
defmacro get(path, handler) do
quote do: @routes {:get, unquote(path), unquote(handler)}
end
defmacro __before_compile__(env) do
for {verb, path, handler} <- Module.get_attribute(env.module, :routes) do
quote do
def dispatch(unquote(verb), unquote(path)), do: unquote(handler).()
end
end
end
end
defmodule MyApp do
use Router
get "/health", fn -> :ok end
get "/users", fn -> list_users() end
end
Every route becomes its own function head, written into the module before the module has finished compiling. Dispatch is then pattern matching on the BEAM, with no lookup table between the request and the work. This is roughly how the Phoenix router does it, and once you have seen the tuples it stops looking like magic and starts looking like filing, which is what infrastructure always turns out to be underneath.
R2 has the other quality as well. Nobody can read his output, and the ship keeps flying.
Luke Skywalker is Nim
Prophecies of this sort always involve somebody from an agricultural planet, largely because nobody in a capital city has the time.
Nim runs your actual language during compilation. Not a template sublanguage, not a pattern matcher shuffling tokens about, but the real thing, executed by a virtual machine inside the compiler, working on a tree that has already been through the type checker and therefore knows what everything is.
import macros, strutils
type User = object
id: int
email: string
active: bool
macro insertFor(T: typedesc, table: static string): string =
var cols, slots: seq[string]
for field in T.getTypeImpl[1].getTypeImpl[2]:
cols.add $field[0]
slots.add "$" & $cols.len
newLit("INSERT INTO " & table & " (" & cols.join(", ") &
") VALUES (" & slots.join(", ") & ")")
const q = insertFor(User, "users")
# INSERT INTO users (id, email, active) VALUES ($1, $2, $3)
The macro is handed the type itself rather than any example of it, and walks the field list while the compiler is still thinking things over. What comes back is a const, so the statement sits in the binary with nothing left to work out at startup.
Now add a column to User and every statement built this way changes on the next build, with nobody editing a query. Take a column away and the code that used it stops compiling, on the line that used it, long before anything gets near a database. No ORM, no annotations, no runtime reflection, no generated files waiting for somebody to remember them. It is C-3PO's job description, carried out inside the compiler by a droid who does not talk.
Jar Jar Binks, and how it actually happens
Vader gets the credit. Vader is only the invoice.
The Republic was not overthrown. It was voted away by a well meaning idiot who stood up and proposed that the Chancellor be granted emergency powers, because it seemed helpful at the time and everybody was tired.
Your codebase has one of these. A convenient little macro, added on a Friday because it saved eight lines and everyone agreed it was rather neat. Eighteen months later half the system routes through it, its behaviour has acquired three special cases nobody wrote down, and nobody remembers voting for any of it. It was never debated. It was accepted, because at the time it was obviously helpful.
What you end up with is the Vader codebase: more generated than written, technically alive, unpleasant to stand near, maintained by two people who stopped explaining it to anyone.
The point of the hilt
The saber is elegant because a competent person built it for their own hand. Handed to somebody who has not done the work, it removes a limb, and in practice nearly always their own.
Two habits keep me on the right side of it. Write the repetitive version first, more than once, until the shape stops moving about, because an abstraction laid over a moving target is only a faster way of arriving at the wrong answer. And take a production stack trace from your beautiful new language and hand it to somebody who joined last week. If they can walk from the error message to the line that caused it without a guided tour, you have built a lightsaber. If they cannot, you have built something that will be maintained out of superstition long after the last person who understood it has left for a job with fewer parentheses.
The Force was never the problem. It has never once been the problem.
So: which macro in your codebase is everybody quietly afraid of, and which of this lot wrote it?
Notes
[1] There is presumably paperwork as well. There is always paperwork. Somewhere in the Temple was a Jedi whose entire career consisted of crystal requisitions, and he did not get a film.
[2] The confusion arrives later, at the darts final, when the pub catches fire.
[3] Detractors maintain it stands for Lots of Irritating Superfluous Parentheses. Lisp programmers have heard this. Lisp programmers were hearing this before the detractors' parents were introduced.
[4] It is always a Sunday and it is always payments. Nobody knows why. There are theories.