Compare two arms cell by cell

Two arms scored on the same held-out units do not necessarily have the same set of defined cells, so a difference of two marginal means is not a difference between the arms. This takes the difference inside each (variable, fold) cell both arms scored, averages it within a variable over its folds, and summarises those per-variable means, the variables being the independent replicates.

Usage

paired_contrast(ladder, a, b, interval = c("variables", "nested_cv"))

Arguments

ladder

A grain_ladder() result.

a, b

The two arms, each named "grain|learner".

interval

Which interval the row carries. "variables" is the spread across the response variables of this dataset. "nested_cv" is an interval for the difference in the two arms' risk on a new sample, by the nested cross-validation of Bates, Hastie and Tibshirani (2024) read on the difference of the two arms' cell scores, which needs a ladder fitted with grain_ladder(interval = "nested_cv"). The paper gives the estimator for one procedure's error and names the difference between two as an extension; the identity its estimator rests on is on the difference of two losses as much as on one.

Value

A one-row data frame: the mean per-variable difference; the centre of the interval and its bounds, on Student's t across variables or on nested cross-validation as interval names, which the row carries; the number of variables the difference favours; the paired cells and variables it rests on; a Wilcoxon signed-rank p-value; and p_method, "exact" where the p-value is read off the exact distribution and "normal" where it is the normal approximation with continuity and tie corrections, which it is when the per-variable differences hold a zero or a tie or number fifty or more.

Details

Pairing also cancels what a threshold-selected metric carries in its level. TSS read at the threshold that maximises it is biased upward where presences are thin, both arms carry the same bias on the same cell, and it cancels in the difference. That is why the levels a ladder reports are upper bounds while the differences between arms are read at face value.

An arm is named whole, by its grain and its learner. A learner named alone would have to take its best grain, and that grain is chosen on the held-out scores the contrast is then read off: the difference becomes one between two maxima, favouring whichever learner ran across more grains, and neither the interval nor the p-value accounts for the choice. It is the mechanism tss_inflation() measures one level down, and here pairing does not cancel it. select_grain() chooses a grain on inner folds instead, and its compare argument contrasts the selection with the arms of a ladder on matched cells.

References

Bates, S., Hastie, T. and Tibshirani, R. (2024). Cross-validation: what does it estimate and how well does it do it? Journal of the American Statistical Association 119(546), 1434-1445. doi:10.1080/01621459.2023.2197686

Examples

set.seed(1)
t <- seq(as.POSIXct("2021-09-01", tz = "UTC"), by = "hour", length.out = 24 * 200)
units <- sprintf("p%02d", 1:60)
warmth <- rnorm(60)
d <- data.frame(
  plot = rep(units, each = length(t)), t = rep(t, length(units)),
  temp = as.numeric(vapply(warmth, function(w) w + sin(seq_along(t) / 300) + rnorm(length(t)),
                           numeric(length(t)))))
y <- matrix(rbinom(120, 1, plogis(c(warmth, -warmth))), nrow = 60,
            dimnames = list(units, c("sp1", "sp2")))
x <- grain_matrix(d, plot, t, temp, grain = c("week", "month"))
lad <- grain_ladder(x, y, elasticnet(), folds = fold_map(y, v = 3), verbose = FALSE)
paired_contrast(lad, "week|elasticnet", "month|elasticnet")