Skip to contents

What part of the record a fitted model reads

Source: R/occlusion.R, R/report.R
occlusion.Rd

Holds one bin of the record back at a time, rescores the held-out units, and records the fall in score as that bin's weight. Nothing is refitted: the models kept by grain_ladder(keep_fits = TRUE) are the ones read, so the profile describes the models that produced the reported scores rather than a fresh set of them.

Usage

occlusion(x, ...)

# Default S3 method
occlusion(x, ...)

# S3 method for class 'timesift_ladder'
occlusion(
  x,
  data,
  y,
  arm,
  over = c("bin", "channel"),
  substitute = c("permute", "fold_mean", "unit_mean"),
  metric = NULL,
  permutations = 20L,
  seed = 1L,
  ...
)

# S3 method for class 'timesift'
occlusion(x, candidate, over = c("bin", "channel"), ...)

Arguments

x

A timesift() fit, or a grain_ladder() result, in either case fitted with keep_fits = TRUE.

...

Passed to the method.

data

The representation set the ladder was fitted on.

y

The response it was fitted to.

arm

The arm to read, as "grain|learner" or "learner".

over

"bin" to hold each bin back in turn, "channel" for each channel.

substitute

What a held-back part is replaced by: "permute", "fold_mean" or "unit_mean".

metric

Name of the registered metric the rescoring is read by, or a function of (y, p). Left unset it is the one the fit was scored under, so a weight is a fall in the number summary() reports rather than in a second one. "roc_auc" is usually the steadier reading over many rescorings: it responds to every reordering of the units, where a maximum over thresholds frequently does not move at all.

permutations

Draws averaged over, for substitute = "permute".

seed

Random seed.

candidate

Name of the candidate to read, for a run, or "ensemble" for the combination the run fitted. The members of an ensemble read different bins, so the ensemble is held back by channel: a channel is withheld from every member that carries it, all of them shifted the same way, and the members' predictions are combined by the stack the run fitted.

Value

A data frame of one row per held-back part and variable, carrying the mean weight over folds, the score with and without the part, and importance: one minus the correlation between the predictions with and without the part, which is how biomod2 scores a predictor and needs no response. Both are read on the cells the fit was scored on, and importance is NA where either set of predictions is constant.

Details

A model has to be shown something in place of a held-back bin, and what it is shown decides what the weight means. Permuting the bin's values across units keeps the observed readings exactly and cuts only the link between a reading and its unit. Replacing every unit by the fitting-fold mean removes all between-unit variation while keeping the shape of the year. Replacing the bin by each unit's own mean over the record keeps how warm a unit is and removes only that bin's departure from it.

A bin is held back whole: a permutation moves every channel of the bin together, so a unit is shown another unit's week rather than a coldest day from one unit beside a warmest day from another. A channel that is the same for every unit, as calendar_channels() are, says where the bin sits rather than what a unit read there, and is left in place.

Read with over = "channel" the same machinery asks what each statistic of a grain carries, holding one channel back across the whole record instead of one bin across all channels.

A column named in static is a channel of a representation built from a record, constant across its bins, so over = "channel" holds it back; where the run has no record the columns are the bins of one block and over = "bin" does.

Reached through a timesift() run rather than through a ladder, the profile reads the per-fold models the run was told to keep, so every bin is held back from a model that never saw the units it is rescored on. The candidate is named as summary() reports it.

Examples

set.seed(1)
t <- seq(as.POSIXct("2021-09-01", tz = "UTC"), by = "hour", length.out = 24 * 120)
units <- sprintf("p%02d", 1:40)
warmth <- rnorm(40)
d <- data.frame(
  plot = rep(units, each = length(t)), t = rep(t, length(units)),
  temp = as.numeric(vapply(warmth, function(w) w + sin(seq_along(t) / 300) + rnorm(length(t)),
                           numeric(length(t)))))
y <- matrix(rbinom(80, 1, plogis(c(warmth, -warmth))), nrow = 40,
            dimnames = list(units, c("sp1", "sp2")))
x <- grain_matrix(d, plot, t, temp, grain = "month")
lad <- grain_ladder(x, y, elasticnet(), folds = fold_map(y, v = 3),
                     keep_fits = TRUE, verbose = FALSE)
head(occlusion(lad, x, y, "month|elasticnet", permutations = 3))