Fit at every grain and see where skill saturates

Cross-validates every learner at every grain of a representation set, on one fold map and one mask of scorable cells, and returns the score of each (grain, learner, variable, fold) cell. It is the measurement the package exists for: how much of a record a model needs, read off the point where making the record finer stops paying.

Usage

grain_ladder(
  x,
  y,
  learners,
  folds = NULL,
  response = "presence_absence",
  metric = NULL,
  control = train_control(),
  keep_fits = FALSE,
  interval = c("variables", "nested_cv"),
  repeats = 1L,
  seed = 1L,
  verbose = TRUE
)

# S3 method for class 'timesift_ladder'
summary(object, ...)

Arguments

x: A grain_matrix() result, a timesift_set(), or a named list of representations.
y: The response for the same units.
learners: A learner, a set of them from c(), a list, or names of registered ones. An unnamed set is labelled by each learner's own name.
folds: A fold map from fold_map(), or any named integer vector. Built with the defaults of fold_map() when not given.
response: Name of the registered response head.
metric: Name of a registered metric, or a function of (y, p), or NULL for the response head's own. Whichever it is, it travels with the fit and is what every later rescoring reads; a function is reported as <function>.
control: train_control(), the training settings every neural learner of the ladder reads. A learner carrying a control of its own overrides it on the settings that control names.
keep_fits: Keep every per-fold fitted model, which is what lets occlusion() read a fitted model without refitting it.
interval: "nested_cv" refits every arm inside every outer training set, which is what paired_contrast() reads an interval for the difference in risk off. "variables", the default, fits nothing further, and a contrast is then read across the response variables.
repeats: Repetitions of the nested cross-validation, each on its own fold map. The first is the map the ladder was cross-validated on.
seed: Seed the repetitions' fold maps are drawn under. Two tables whose contrast is to be read take the same folds, repeats and seed.
verbose: Report each arm and each fold as it runs.
object: A ladder.
...: Ignored, so that summary() takes the arguments its generic declares.

Value

A data frame of one row per scored cell, of class timesift_ladder, carrying the grain, the learner, the variable, the fold and the score. The held-out prediction of every unit is kept in the predictions attribute, and the scorable-cell mask in cells.

Details

Every arm sees identical splits and is restricted to identical cells, so the arms' means share a denominator and any two of them can be compared cell by cell with paired_contrast().

Examples

set.seed(1)
t <- seq(as.POSIXct("2021-09-01", tz = "UTC"), by = "hour", length.out = 24 * 200)
units <- sprintf("p%02d", 1:60)
warmth <- rnorm(60)
d <- data.frame(
  plot = rep(units, each = length(t)), t = rep(t, length(units)),
  temp = as.numeric(vapply(warmth, function(w) w + sin(seq_along(t) / 300) + rnorm(length(t)),
                           numeric(length(t)))))
y <- matrix(rbinom(120, 1, plogis(c(warmth, -warmth))), nrow = 60,
            dimnames = list(units, c("sp1", "sp2")))
x <- grain_matrix(d, plot, t, temp, grain = c("week", "month"))
lad <- grain_ladder(x, y, elasticnet(), folds = fold_map(y, v = 3), verbose = FALSE)
summary(lad)