Compare two arms cell by cell

Two arms scored on the same held-out units do not necessarily have the same set of defined cells, so a difference of two marginal means is not a difference between the arms. This takes the difference inside each (variable, fold) cell both arms scored, averages it within a variable over its folds, and summarises those per-variable means, the variables being the independent replicates.

Usage

paired_contrast(ladder, a, b, interval = c("variables", "nested_cv"))

Arguments

ladder

A grain_ladder() result.

a, b

The two arms, each named "grain|learner".

interval

Which interval the row carries. "variables" is the spread across the response variables of this dataset. "nested_cv" is an interval for the difference in the two arms' risk on a new sample, by the nested cross-validation of Bates, Hastie and Tibshirani (2024) read on the difference of the two arms' cell scores, which needs a ladder fitted with grain_ladder(interval = "nested_cv"). The paper gives the estimator for one procedure's error and names the difference between two as an extension; the identity its estimator rests on is on the difference of two losses as much as on one.

Value

A one-row data frame: the mean per-variable difference; the centre of the interval and its bounds, on Student's t across variables or on nested cross-validation as interval names, which the row carries; the number of variables the difference favours; the paired cells and variables it rests on; a Wilcoxon signed-rank p-value; and p_method, "exact" where the p-value is read off the exact distribution and "normal" where it is the normal approximation with continuity and tie corrections, which it is when the per-variable differences hold a zero or a tie or number fifty or more.

Details

Pairing removes the variation between variables and the part of a threshold-selected metric's bias that the design sets, but not the part that belongs to each arm. TSS read at the threshold that maximises it is biased upward where presences are thin, and how far depends on how an arm's predictions are distributed as well as on how many presences the cell holds, so two arms of equal skill on the same cell can carry different biases. A difference in TSS can therefore favour one arm with no difference in skill behind it. A threshold-free metric such as roc_auc() has no cut to choose, and a TSS contrast is best read beside the same contrast under it.

An arm is named whole, by its grain and its learner. A learner named alone would have to take its best grain, and that grain is chosen on the held-out scores the contrast is then read off: the difference becomes one between two maxima, favouring whichever learner ran across more grains, and neither the interval nor the p-value accounts for the choice. It is the mechanism tss_inflation() measures one level down, and here pairing does not cancel it. select_grain() chooses a grain on inner folds instead, and its compare argument contrasts the selection with the arms of a ladder on matched cells.

References

Bates, S., Hastie, T. and Tibshirani, R. (2024). Cross-validation: what does it estimate and how well does it do it? Journal of the American Statistical Association 119(546), 1434-1445. doi:10.1080/01621459.2023.2197686

Examples

set.seed(1)
t <- seq(as.POSIXct("2021-09-01", tz = "UTC"), by = "hour", length.out = 24 * 200)
units <- sprintf("p%02d", 1:60)
warmth <- rnorm(60)
d <- data.frame(
  plot = rep(units, each = length(t)), t = rep(t, length(units)),
  temp = as.numeric(vapply(warmth, function(w) w + sin(seq_along(t) / 300) + rnorm(length(t)),
                           numeric(length(t)))))
y <- matrix(rbinom(120, 1, plogis(c(warmth, -warmth))), nrow = 60,
            dimnames = list(units, c("sp1", "sp2")))
x <- grain_matrix(d, plot, t, temp, grain = c("week", "month"))
lad <- grain_ladder(x, y, elasticnet(), folds = fold_map(y, v = 3), verbose = FALSE)
paired_contrast(lad, "week|elasticnet", "month|elasticnet")