19 Iteration without loops

You have a list of three data frames and you want the mean of one column in each. You can write six lines of for loop, or you can hand the computation to a function that does the looping for you. Section 7.1 showed functions being passed as arguments; this chapter is what that buys you.

This chapter uses three packages. Load them now if you are following along:

library(palmerpenguins)
library(purrr)
library(dplyr)

We will work with the penguins data split by species throughout:

penguins_split <- split(penguins, penguins$species)
names(penguins_split)
#> [1] "Adelie"    "Chinstrap" "Gentoo"

Three data frames, one per species. Almost everything in this chapter amounts to doing something to each of them, and the question is always the same: how much of the work should you spell out yourself, and how much should the language handle?

19.1 The problem with `for` loops

Suppose you want the mean body mass for each species. Here is the loop version:

results <- vector("list", length(penguins_split))
for (i in seq_along(penguins_split)) {
  results[[i]] <- mean(penguins_split[[i]]$body_mass_g, na.rm = TRUE)
}
names(results) <- names(penguins_split)
results
#> $Adelie
#> [1] 3700.662
#> 
#> $Chinstrap
#> [1] 3733.088
#> 
#> $Gentoo
#> [1] 5076.016

Six lines to say “compute the mean mass for each group.” Look at how much of that is bookkeeping: pre-allocate a container, set up an index, store results at the right position, copy names over at the end. The actual computation, mean(df$body_mass_g, na.rm = TRUE), sits buried in the middle, surrounded by scaffolding that has nothing to do with the question you asked.

A functional says the same thing in one line:

map_dbl(penguins_split, \(df) mean(df$body_mass_g, na.rm = TRUE))
#>    Adelie Chinstrap    Gentoo 
#>  3700.662  3733.088  5076.016

Same answer. No index variable, no pre-allocation, no name copying. You said what to compute and map_dbl() handled the iteration. A function that takes another function and applies it across a collection like this is called a functional.

A well-written loop in modern R runs as fast as the functional. What the loop mixed together was the mechanism of iteration (allocate, index, store) and the content of the computation (compute a mean). The functional keeps the mechanism inside itself and leaves the content in the function you pass, and once the two are apart, each is easier to read, test, and reuse on its own.

Opinion

For loops are verbose R. Once the same scaffolding appears three or four times in a project, a functional replaces all of it with one line.

Exercises

Write a for loop that computes the number of rows in each element of penguins_split. Then rewrite it using map_int() and nrow.
What happens if you forget to pre-allocate results and instead grow it with results <- c(results, value) inside a loop? Why is this slower?

19.2 `lapply()` and `sapply()`: base R functionals

lapply() is base R’s version of the same idea:

lapply(penguins_split, \(df) mean(df$body_mass_g, na.rm = TRUE))
#> $Adelie
#> [1] 3700.662
#> 
#> $Chinstrap
#> [1] 3733.088
#> 
#> $Gentoo
#> [1] 5076.016

lapply(x, f) applies f to each element of x and returns a list. Always a list. That predictability is its strength.

Where map comes from

Handing a function to something that applies it to every element of a list is older than R and older than S. John McCarthy’s 1960 paper on Lisp defined maplist, which applies a function to each successive tail of a list, and the LISP 1.5 manual added mapcar, which applies it to each element. Later languages (Scheme, ML, Haskell) settled on the name map. R inherited the idea twice: once as lapply() in base R, and again as purrr::map() in the tidyverse.

sapply() does the same thing, but tries to “simplify” the result:

sapply(penguins_split, \(df) mean(df$body_mass_g, na.rm = TRUE))
#>    Adelie Chinstrap    Gentoo 
#>  3700.662  3733.088  5076.016

When every call to f returns a single value, sapply() hands you a named vector. But if f returns different lengths for different elements, sapply() falls back to a list. You cannot predict the return type without knowing the output of f for every element of x. Fine at the console; unreliable inside a function that other code depends on.

vapply() solves this by making you declare the expected type:

vapply(penguins_split, \(df) mean(df$body_mass_g, na.rm = TRUE), numeric(1))
#>    Adelie Chinstrap    Gentoo 
#>  3700.662  3733.088  5076.016

The third argument, numeric(1), says “I expect each call to return a single numeric value.” If any call returns something else, you get an error instead of a silent type change.

Two other base R functionals are worth knowing. tapply(x, group, f) applies f to subsets of x defined by group:

tapply(penguins$body_mass_g, penguins$species, mean, na.rm = TRUE)
#>    Adelie Chinstrap    Gentoo 
#>  3700.662  3733.088  5076.016

And mapply(f, x, y) iterates over multiple inputs in parallel:

mapply(paste, c("one", "two", "three"), c("fish", "cat", "bird"))
#>          one          two        three 
#>   "one fish"    "two cat" "three bird"

Opinion

Use lapply() when you want a list. Use vapply() when you want a vector and want safety. Never use sapply() in non-interactive code: its return type depends on the data, and that is a bug waiting in your pipeline.

Exercises

Use lapply() and nrow to get the number of rows in each element of penguins_split.
Use vapply() to compute the median flipper length for each species. Declare the expected output type.
What does sapply(list(), mean) return? What about vapply(list(), mean, numeric(1))? Which is safer?

19.3 `purrr::map()`: the tidyverse functional

The purrr package provides map(), which does the same thing as lapply():

map(penguins_split, \(df) mean(df$body_mass_g, na.rm = TRUE))
#> $Adelie
#> [1] 3700.662
#> 
#> $Chinstrap
#> [1] 3733.088
#> 
#> $Gentoo
#> [1] 5076.016

Apply a function to each element, return a list. So why bother with a new package?

The typed variants. map_dbl() returns a double vector, map_chr() a character vector, and if the function returns the wrong type you get an error where sapply() would have handed you a list:

map_dbl(penguins_split, \(df) mean(df$body_mass_g, na.rm = TRUE))
#>    Adelie Chinstrap    Gentoo 
#>  3700.662  3733.088  5076.016

Like lapply(), map() takes the collection first, so it sits naturally after a pipe:

penguins_split |>
  map_dbl(\(df) mean(df$body_mass_g, na.rm = TRUE))
#>    Adelie Chinstrap    Gentoo 
#>  3700.662  3733.088  5076.016

You may encounter the formula shorthand ~ mean(.x$body_mass_g, na.rm = TRUE) in older code, purrr’s anonymous-function syntax from before R 4.1 introduced \(x). Prefer \(x) in new code.

map() always returns a list and map_dbl() always returns a double vector, whatever the data. In a long pipeline that is what lets you know the shape of every intermediate result without running it. And the list has three elements, so the function runs three times: the structure of the data determines how many applications occur, and you stop managing the iteration yourself.

Functors and the shape of iteration

map() embodies a pattern from category theory: the functor. map(xs, f) applies f to each element of xs, preserving the container structure (a list of three elements in, a list of three elements out). A functor lifts a function between values (f :: A -> B) to a function between containers (map(f) :: List(A) -> List(B)). You do not need the theory to use map(), but recognizing the pattern explains why map() feels the same whether you apply it to a list, a vector, or a set of data frame columns with across(): it is the same structural idea in different containers.

Exercises

Use map_dbl() to compute the standard deviation of body_mass_g for each species in penguins_split.
Use map_chr() to extract the first value of species from each element of penguins_split. (Hint: \(df) as.character(df$species[1]).)
Rewrite the for loop from Section 19.1 using map_dbl(). Compare the two versions for readability.

19.4 Typed variants

map_int() and map_lgl() complete the set:

map_int(penguins_split, nrow)
#>    Adelie Chinstrap    Gentoo 
#>       152        68       124

map_lgl(penguins_split, \(df) any(is.na(df$body_mass_g)))
#>    Adelie Chinstrap    Gentoo 
#>      TRUE     FALSE      TRUE

Each typed variant checks both the type and the length of every result: map_dbl() wants a single double per call, map_chr() a single string, map_lgl() a single logical, map_int() a single integer. Anything else is an error:

map_dbl(penguins_split, \(df) df$body_mass_g)
#> Error in `map_dbl()`:
#> ℹ In index: 1.
#> ℹ With name: Adelie.
#> Caused by error:
#> ! Result must be length 1, not 152.

When the output does not match your expectation, you find out immediately, not three functions downstream when something produces NULL instead of a number.

For results that are data frames, the current pattern is map() followed by list_rbind(); the older map_dfr() still works but is superseded:

penguins_split |>
  map(\(df) tibble(
    species = df$species[1],
    mean_mass = mean(df$body_mass_g, na.rm = TRUE),
    n = nrow(df)
  )) |>
  list_rbind()
#> # A tibble: 3 × 3
#>   species   mean_mass     n
#>   <fct>         <dbl> <int>
#> 1 Adelie        3701.   152
#> 2 Chinstrap     3733.    68
#> 3 Gentoo        5076.   124

map() returns a list of data frames, and list_rbind() stacks them into one. But what about iterating over more than one thing at a time?

Exercises

Use map_int() and nrow to count the rows in each element of penguins_split.
Use map_lgl() to check which elements of penguins_split have more than 100 rows.
Use map() and list_rbind() to build a summary data frame with one row per species, containing the species name, median bill length, and median bill depth.

19.5 Iterating over multiple inputs

Sometimes you need to walk over two inputs in lockstep, pulling one element from each on every step. map2() handles this:

species_names <- names(penguins_split)
species_counts <- map_int(penguins_split, nrow)

map2_chr(species_names, species_counts, \(name, n) paste(name, ":", n, "penguins"))
#> [1] "Adelie : 152 penguins"   "Chinstrap : 68 penguins"
#> [3] "Gentoo : 124 penguins"

map2(.x, .y, .f) calls .f(x[[1]], y[[1]]), then .f(x[[2]], y[[2]]), and so on. The two inputs are consumed in parallel, not in a nested grid.

For three or more inputs, use pmap(). You supply a list of vectors (or a data frame, since a data frame is a list of columns):

params <- list(
  mean = c(0, 5, -3),
  sd = c(1, 2, 0.5),
  n = c(3, 3, 3)
)

set.seed(42)
pmap(params, \(mean, sd, n) rnorm(n, mean, sd))
#> [[1]]
#> [1]  1.3709584 -0.5646982  0.3631284
#> 
#> [[2]]
#> [1] 6.265725 5.808537 4.787751
#> 
#> [[3]]
#> [1] -2.244239 -3.047330 -1.990788

Each row of parameters becomes one call. pmap() is the general case; map2() is the special case for exactly two inputs.

The base R equivalents are Map(f, x, y) (which returns a list, like map2()) and mapply(f, x, y) (which simplifies, like sapply()). The same warning applies: mapply is unpredictable in return type.

pmap() earns its place when a table of parameters drives the work: different models on different subsets with different formulas, all in one tibble. Suppose you want to regress body mass on flipper length for each species, but with a different formula for each:

configs <- tibble(
  data = penguins_split,
  formula = list(
    body_mass_g ~ flipper_length_mm,
    body_mass_g ~ flipper_length_mm + bill_length_mm,
    body_mass_g ~ flipper_length_mm * bill_length_mm
  )
)

models <- pmap(configs, \(data, formula) lm(formula, data = data))
map_dbl(models, \(m) summary(m)$r.squared)
#>    Adelie Chinstrap    Gentoo 
#> 0.2192128 0.4688941 0.5923485

pmap() iterates over the rows of the configuration table, fitting one model per species. The entire experiment lives in one tibble, and adding a fourth species or a fourth formula means adding a row, not rewriting the loop.

Exercises

Use map2_chr() to paste together the vectors c("one", "two", "three") and c("fish", "cat", "bird") with a space in between.
Create a list of three means and three standard deviations. Use map2_dbl() to generate one random normal value for each pair. (Hint: \(m, s) rnorm(1, m, s).)
A data frame is a list of columns. What does pmap_dbl(data.frame(x = 1:3, y = 4:6), \(x, y) x + y) return?

19.6 Side effects: `walk()`

Everything so far has been about functions that compute something: they take input and produce output. But some functions exist for what they do, not what they return: printing a message, writing a file, drawing a plot. For these, use walk().

walk(names(penguins_split), \(name) cat(name, "\n"))
#> Adelie 
#> Chinstrap 
#> Gentoo

walk(.x, .f) applies .f to each element of .x, but instead of collecting results into a list, it returns .x invisibly. The point is the side effect, not the return value.

The two-input version, walk2(), is especially useful for writing files:

file_paths <- paste0("data/", names(penguins_split), ".csv")
walk2(penguins_split, file_paths, \(df, path) write.csv(df, path, row.names = FALSE))

Each data frame gets written to its own file. walk2() iterates over the data frames and paths in parallel, calling write.csv() for each pair.

Use walk() whenever .f does something rather than computes something. If you use map() for side effects, you end up with a list of NULL values cluttering your console, which is the language telling you that you reached for the wrong tool.

Because walk() returns its input invisibly, it slots cleanly into the middle of a pipeline:

penguins_split |>
  walk(\(df) cat("Species:", as.character(df$species[1]), "- n =", nrow(df), "\n")) |>
  map_dbl(\(df) mean(df$body_mass_g, na.rm = TRUE))
#> Species: Adelie - n = 152 
#> Species: Chinstrap - n = 68 
#> Species: Gentoo - n = 124
#>    Adelie Chinstrap    Gentoo 
#>  3700.662  3733.088  5076.016

The walk() prints a line for each species, then passes penguins_split through unchanged to map_dbl(). Side effects and computation in one pipeline. One kind of iteration, though, is still out of reach for every functional so far.

Exercises

Use walk() to print the column names of each element of penguins_split. (They should all be the same.)
Use walk2() and cat() to print lines like "Adelie has 152 rows" for each species.

19.7 When loops are fine

Not everything should be a functional. Here is a random walk, where each step depends on the one before it:

x <- numeric(20)
x[1] <- 0
for (i in 2:20) {
  x[i] <- x[i - 1] + rnorm(1)
}
x
#>  [1]  0.0000000 -0.0627141  1.2421556  3.5288009  2.1399402  1.8611515
#>  [7]  1.7278301  2.3637805  2.0795276 -0.5769278 -3.0173947 -1.6972814
#> [13] -2.0039200 -3.7852284 -3.9571458 -2.7424711 -0.8472776 -1.2777467
#> [19] -1.5350161 -3.2981792

map() cannot write this, because it hands each element to the function on its own and assumes the elements are independent. Whenever iteration i needs the result of iteration i - 1 (simulation chains, iterative algorithms, anything that carries state), the loop is the honest tool. purrr’s reduce() and accumulate() handle some of these sequential cases, and Chapter 21 covers them.

A loop is also clearer when the body is complex. next to skip elements, break to stop early, several branches that depend on runtime state: contorting all of that into a function passed to map() costs more than it saves.

And a loop you understand is better than a functional you do not. Use map() when the pattern clicks; until then, write the loop, get the right answer, and refactor later when the shape of the problem becomes obvious.

Opinion

Replace a loop with a functional when the pattern is “do this to each thing independently.” Keep a loop when iterations talk to each other.

19.8 `across()` revisited

In Section 14.7 you used across() to apply a function to several columns inside a dplyr verb:

penguins |>
  summarise(across(where(is.numeric), \(x) mean(x, na.rm = TRUE)))
#> # A tibble: 1 × 5
#>   bill_length_mm bill_depth_mm flipper_length_mm body_mass_g  year
#>            <dbl>         <dbl>             <dbl>       <dbl> <dbl>
#> 1           43.9          17.2              201.       4202. 2008.

across(.cols, .fns) takes a column selection and a function (or a list of functions) and applies the function to each selected column. That is map() over the columns of a data frame, specialized for the inside of a dplyr verb.

Multiple functions at once:

penguins |>
  summarise(across(
    where(is.numeric),
    list(mean = \(x) mean(x, na.rm = TRUE), sd = \(x) sd(x, na.rm = TRUE))
  ))
#> # A tibble: 1 × 10
#>   bill_length_mm_mean bill_length_mm_sd bill_depth_mm_mean bill_depth_mm_sd
#>                 <dbl>             <dbl>              <dbl>            <dbl>
#> 1                43.9              5.46               17.2             1.97
#> # ℹ 6 more variables: flipper_length_mm_mean <dbl>,
#> #   flipper_length_mm_sd <dbl>, body_mass_g_mean <dbl>,
#> #   body_mass_g_sd <dbl>, year_mean <dbl>, year_sd <dbl>

Combined with group_by():

penguins |>
  group_by(species) |>
  summarise(across(where(is.numeric), \(x) mean(x, na.rm = TRUE)))
#> # A tibble: 3 × 6
#>   species   bill_length_mm bill_depth_mm flipper_length_mm body_mass_g  year
#>   <fct>              <dbl>         <dbl>             <dbl>       <dbl> <dbl>
#> 1 Adelie              38.8          18.3              190.       3701. 2008.
#> 2 Chinstrap           48.8          18.4              196.       3733. 2008.
#> 3 Gentoo              47.5          15.0              217.       5076. 2008.

across() replaces the older summarise_at(), summarise_if(), and summarise_all(), which are now superseded. If you encounter them in older code, they do the same thing with less flexible syntax.

Under the hood, across() iterates over the selected columns, applies the function to each one, and assembles the results. That is the same thing map() does over list entries and sapply() does over vector values. The for loop at the start of this chapter, lapply(), map_dbl(), across(): each separates what to do from how to iterate, and each walks a different container with the same logic.

Exercises

Use across() inside summarise() to compute the median() of every numeric column in penguins, removing NA values.
Use across() with a list of two functions to compute both the min() and max() of every numeric column, grouped by species.
What does across(where(is.character), n_distinct) compute when used inside summarise()? Try it on penguins.

19.1 The problem with for loops

Exercises

19.2 lapply() and sapply(): base R functionals

Exercises

19.3 purrr::map(): the tidyverse functional

Exercises

19.4 Typed variants

Exercises

19.5 Iterating over multiple inputs

Exercises

19.6 Side effects: walk()

Exercises

19.7 When loops are fine

19.8 across() revisited

Exercises

19.1 The problem with `for` loops

19.2 `lapply()` and `sapply()`: base R functionals

19.3 `purrr::map()`: the tidyverse functional

19.6 Side effects: `walk()`

19.8 `across()` revisited