26 Metaprogramming

You have been writing lm(y ~ x) and filter(penguins, species == "Adelie") for chapters. Both do something that might seem impossible: y ~ x is not evaluated before lm() sees it, and species is found inside a data frame, not in your environment. Normally, 1 + 1 evaluates to 2 and the expression disappears. R can keep it. Code in R is a data structure you can hold, examine, rearrange, and run whenever you choose.

26.1 Code is data

quote() captures an expression without evaluating it:

quote(x + 1)
#> x + 1

No result. What you got back is the expression itself: x + 1 was not computed but returned as a data structure (a call object) that you can store, inspect, and eventually evaluate. quote(1 + 1) gives you the expression tree, not the value 2.

e <- quote(x + 1)
typeof(e)
#> [1] "language"
class(e)
#> [1] "call"

The rlang package provides expr(), which does the same thing:

rlang::expr(x + 1)
#> x + 1

To run a captured expression, use eval():

x <- 10
eval(e)
#> [1] 11

eval() takes the frozen expression and evaluates it in the current environment, where x is 10, so the result is 11. The cycle is always the same: capture code, possibly modify it, then evaluate it somewhere. Code that operates on code in this way is called metaprogramming, and you have been using it since your first formula, your first aes(), and your first dplyr pipeline.

In Section 7.5, you saw that + is a function and 2 + 3 is a function call. That function call is also an ordinary R object. It lives in memory, can be subset with [[, modified by assignment, passed to functions like any vector or list, and evaluated in any environment you choose. Tidy evaluation, formula interfaces, and ggplot2’s aes() all rest on that.

Where quote comes from

In lambda calculus, (λx. x + 1)(3) reduces to 4; quoting the term gives you the syntactic object (λx. x + 1)(3) instead, something you can pull apart and study without triggering the reduction (Section 1.2). Lisp built that distinction into the language: (quote (+ 1 2)) returns the list (+ 1 2) instead of computing 3, and since Lisp programs are themselves lists, code and data share one representation. A language with that property is called homoiconic. R has it too: R code is a tree of call objects, and call objects are R data. Python, Java and C++ store code as text, so metaprogramming there starts by parsing a string.

26.2 Abstract syntax trees

Every R expression has a tree structure called an abstract syntax tree (AST). The lobstr package makes them visible:

lobstr::ast(x + y * 2)
#> █─`+` 
#> ├─x 
#> └─█─`*` 
#>   ├─y 
#>   └─2

What looks like a flat sequence of tokens on the page is a tree where + sits at the root, x hangs off one branch, and * (y, 2) hangs off the other. The tree encodes precedence: * binds tighter than +, so y * 2 forms a subtree nested inside the + node, and R evaluates it first without needing parentheses from you.

x and y are names; when the tree is evaluated, R looks up whatever value is bound to them. 2 is a constant and needs no lookup. + and * are function calls, each with its function as the first child and its arguments as the rest. Those three kinds of node, constants, symbols (the technical name for names) and calls, make up every R expression there is. Even control flow is a call:

lobstr::ast(if (x > 0) "yes" else "no")
#> █─`if` 
#> ├─█─`>` 
#> │ ├─x 
#> │ └─0 
#> ├─"yes" 
#> └─"no"

if (x > 0) "yes" else "no" is a call to `if` with three arguments. There is no syntax that escapes the tree.

You can take a call object apart with standard list operations, where the first element is the function and the rest are the arguments:

e <- quote(mean(x, na.rm = TRUE))
e[[1]]
#> mean
e[[2]]
#> x
e[[3]]
#> [1] TRUE

as.list() converts the whole call into a list, which makes the structure easy to see at a glance:

as.list(e)
#> [[1]]
#> mean
#> 
#> [[2]]
#> x
#> 
#> $na.rm
#> [1] TRUE

You can convert between text and expressions with parse() and deparse():

deparse(quote(x + y * 2))
#> [1] "x + y * 2"
parse(text = "x + y * 2")[[1]]
#> x + y * 2

The conversion is not perfectly symmetric; comments and whitespace vanish in the round trip. But the tree structure survives, and the tree is what matters. So what happens when you need to capture not your own expression but someone else’s?

Exercises

Draw the AST for f(a, g(b, c)) on paper. Then check your answer with lobstr::ast().
What does lobstr::ast(1 + 2 + 3) look like? Is + left-associative or right-associative?
Use lobstr::ast() to visualize x[1]. What function is at the root?

26.3 Capturing and evaluating expressions

Capturing your own expression and capturing the caller’s expression require different tools, and confusing the two is a reliable source of bugs.

quote() and expr() capture what you type directly:

quote(a + b)
#> a + b
rlang::expr(a + b)
#> a + b

substitute() captures what the caller passed:

f <- function(x) substitute(x)
f(a + b)
#> a + b

You called f(a + b), and substitute(x) reached back across the function boundary to the call site, grabbing a + b, the expression the caller actually wrote. The same mechanism powers dplyr::filter(): it sees species == "Adelie" as an expression rather than immediately evaluating it and getting an error about a missing variable.

rlang’s version of substitute() is enexpr():

g <- function(x) rlang::enexpr(x)
g(a + b)
#> a + b

Its sibling enquo() captures the expression together with the environment it was written in, and the pair is called a quosure:

h <- function(x) rlang::enquo(x)
h(a + b)
#> <quosure>
#> expr: ^a + b
#> env:  global

If your function will be called alongside dplyr, use the rlang versions, because the rest of tidy evaluation (Section 26.4) is built around them. For standalone base R, substitute() and eval() are sufficient and carry zero dependencies.

Once you have an expression, you evaluate it with eval(). The second argument controls where:

e <- quote(x + 1)
eval(e, list(x = 10))
#> [1] 11
eval(e, list(x = 100))
#> [1] 101

The same frozen expression, evaluated in different environments, gives different results. Data masking (Section 23.4) is exactly that trick: eval_tidy() from rlang evaluates an expression against a data frame, so filter(penguins, species == "Adelie") looks up species in the data rather than in the global environment.

library(rlang)
df <- data.frame(x = c(1, 2, 3), y = c(10, 20, 30))
eval_tidy(expr(x + y), data = df)
#> [1] 11 22 33

One more base R tool worth knowing: match.call(). Inside a function, it returns the entire call as the user typed it, with arguments matched by name:

my_lm <- function(formula, data, subset = NULL) {
  match.call()
}
my_lm(y ~ x, data = mtcars)
#> my_lm(formula = y ~ x, data = mtcars)

Many modeling functions use match.call() to record the call for reproducibility. When you print a fitted model and see Call: lm(formula = y ~ x, data = mtcars), that string came from match.call() stashing the original invocation.

Exercises

Write a function show_code that takes an argument and prints the expression the caller passed (use substitute() and deparse()). Test: show_code(mean(x, na.rm = TRUE)) should print "mean(x, na.rm = TRUE)".
Evaluate quote(x * 2) in an environment where x = 7. Then evaluate it where x = -3.
Write a function that uses match.call() to return its own call. Call it with several arguments and observe the output.

26.4 Building expressions programmatically

What if the function name or the variable is not known until runtime, when your code has to decide at the last moment what expression to build? You need to construct the expression from parts.

rlang::call2() constructs a call object:

rlang::call2("+", 1, 2)
#> 1 + 2
eval(rlang::call2("+", 1, 2))
#> [1] 3

call2() builds one call at a time. For anything larger, write the expression as a template and mark the holes with !! (bang-bang):

my_var <- rlang::expr(body_mass_g)
rlang::expr(mean(!!my_var))
#> mean(body_mass_g)

!! replaced my_var with its value (body_mass_g), producing the expression mean(body_mass_g). Without !!, you would get mean(my_var), which is an entirely different expression and not the one you wanted. A template whose holes are filled at construction time is quasiquotation, and !!! (triple bang, or splice) fills a hole with a whole list of expressions as separate arguments:

vars <- rlang::exprs(species, island)
rlang::expr(group_by(penguins, !!!vars))
#> group_by(penguins, species, island)

This is what { } (embrace) from tidy evaluation does under the hood. When you write a function like:

my_summary <- function(data, var) {
  data |> dplyr::summarise(mean = mean({{ var }}, na.rm = TRUE))
}

my_summary(palmerpenguins::penguins, body_mass_g)
#> # A tibble: 1 × 1
#>    mean
#>   <dbl>
#> 1 4202.
my_summary(palmerpenguins::penguins, flipper_length_mm)
#> # A tibble: 1 × 1
#>    mean
#>   <dbl>
#> 1  201.

the embrace operator defuses var with enquo() and injects it with !!. Tidy evaluation calls the whole cycle defuse-and-inject: capture the caller’s expression, inject it into a template, evaluate the result in the right context, and { } is syntactic sugar for it. The caller writes bare column names, exactly as they would with dplyr directly, and your function forwards them, no quoted strings, no special syntax at the call site.

Base R has bquote() for quasiquotation, using .() instead of !!:

my_var <- quote(body_mass_g)
bquote(mean(.(my_var)))
#> mean(body_mass_g)

It works, but bquote() is less common in practice and does not support splicing. !!, bquote() and expr() all solve the problem Lisp macros have solved since the 1960s: writing code that writes code without falling back to pasting strings together. The expressions you build this way have the same tree structure that quote() gives you by hand.

Exercises

Use rlang::call2() to build the expression sqrt(16), then evaluate it.
Create a variable col <- rlang::expr(bill_length_mm). Use !! to build the expression mean(bill_length_mm, na.rm = TRUE).
Given fns <- rlang::exprs(mean, sd, median), use lapply() and call2() to build three expressions: mean(x), sd(x), median(x).

26.5 Formulas as expressions

Before anyone used the word “metaprogramming” in an R context, there were formulas. When you write:

lm(body_mass_g ~ bill_length_mm, data = palmerpenguins::penguins)
#> 
#> Call:
#> lm(formula = body_mass_g ~ bill_length_mm, data = palmerpenguins::penguins)
#> 
#> Coefficients:
#>    (Intercept)  bill_length_mm  
#>         362.31           87.42

the expression body_mass_g ~ bill_length_mm is not evaluated in the ordinary sense. R captures it as a formula object: two expressions (the left-hand side and the right-hand side) bundled together with the environment where the formula was created.

f <- y ~ x + z
typeof(f)
#> [1] "language"
length(f)
#> [1] 3
f[[2]]
#> y
f[[3]]
#> x + z

A formula stores its terms as call objects. f[[2]] is the left-hand side (y), f[[3]] is the right-hand side (x + z). The formula also carries an environment attribute:

environment(f)
#> <environment: R_GlobalEnv>

Formulas work across function boundaries because the formula remembers where its variables should be looked up. Quosures in tidy evaluation solve the same problem: an expression bundled with its environment.

Where quosures came from

Wickham borrowed the design from formulas, which had been carrying an environment alongside their expressions in R’s modelling functions long before tidy evaluation existed. A quosure still inherits from the formula class: class(rlang::quo(x)) is c("quosure", "formula").

The formula language (+, *, :, -, I()) is a domain-specific language for specifying models. y ~ x1 * x2 does not mean “multiply x1 by x2.” It means “include x1, x2, and their interaction.” model.matrix() interprets these operators to build the design matrix:

model.matrix(~ species + island, data = palmerpenguins::penguins) |> head()
#>   (Intercept) speciesChinstrap speciesGentoo islandDream islandTorgersen
#> 1           1                0             0           0               1
#> 2           1                0             0           0               1
#> 3           1                0             0           0               1
#> 4           1                0             0           0               1
#> 5           1                0             0           0               1
#> 6           1                0             0           0               1

You can also build formulas dynamically, which becomes useful when the set of predictors is not known in advance:

predictors <- c("bill_length_mm", "flipper_length_mm")
f <- as.formula(paste("body_mass_g ~", paste(predictors, collapse = " + ")))
f
#> body_mass_g ~ bill_length_mm + flipper_length_mm

Constructing code from data, then running it: that is metaprogramming at its most practical. The formula is a piece of code assembled from strings, about to be interpreted by lm() as a model specification, and neither the user nor the modeling function needs to know it was built programmatically.

Formulas are non-standard evaluation (Section 23.3): the variable names are unquoted, and body_mass_g is looked up in penguins rather than in the calling environment. dplyr uses the same trick with newer machinery.

Exercises

Create a formula y ~ x1 + x2 and extract its right-hand side.
Build a formula programmatically: given response <- "mpg" and predictors <- c("wt", "hp"), construct the formula mpg ~ wt + hp using as.formula() and paste(). Pass it to lm() with the mtcars dataset.

26.6 When to use metaprogramming

Metaprogramming lets you generate code, build DSLs, and eliminate boilerplate. That power comes with a cost.

It pays when the interface is the product. Model formulas, ggplot2 aesthetics and dplyr pipelines are all DSLs built on metaprogramming, and a package with an interactive interface gains the same concise syntax from the same tools. It pays for code generation: building model specifications programmatically, creating batches of test cases, generating reports from templates. And it pays for inspection, where substitute() shows what was passed and match.call() records the exact call for reproducibility.

It does not pay when an ordinary function would do: if f(x) solves the problem, eval(substitute(...)) only adds a layer to debug. Nor does it make anything faster; metaprogramming is more flexible, which is a different axis. Code that manipulates code is hard to read and hard to debug, so use it when the benefit (a concise user interface) outweighs the cost (a complex implementation), not before.

Opinion

Most R users consume metaprogramming (by using dplyr, ggplot2, formulas) and very few need to produce it. Package authors building interactive interfaces may need these tools; analysts writing data pipelines almost certainly do not. The test: does your user-facing API become meaningfully better with non-standard evaluation? If yes, the complexity is worth absorbing. Saving yourself a quoted string is not enough reason.

26.7 The metaprogramming toolkit

The tools in this chapter fall into two families. Base R has quote(), substitute(), eval(), match.call(), sys.call() and bquote(); they predate rlang and still power much of R’s own infrastructure, and substitute() with eval() is the pair every R programmer should know. rlang’s tidy evaluation adds expr(), enquo(), eval_tidy(), !!, !!! and { }, the system behind dplyr, tidyr and ggplot2. For seeing what you have built, lobstr::ast() draws the tree. Wickham’s Advanced R (second edition) devotes chapters 17 to 21 to metaprogramming, and Mailund’s Metaprogramming in R (Apress, 2017) is a book-length treatment that goes on to domain-specific languages and code generation.

Exercises

Look at the source of dplyr::filter (type dplyr::filter.data.frame at the console). Can you spot where it captures the user’s expressions?
Compare quote(), rlang::expr(), substitute(), and rlang::enexpr(). Write one sentence describing when you would use each.