A chance-corrected agreement rate on a two-by-two table. Read against the observed response it
is a skill score beside tss(); read between two models' decisions on the same units it says
where the two part company.
Usage
kappa_score(y, p, rule = c("prevalence", "youden", "kappa", "mpa"))
decision_threshold(y, ...)
# Default S3 method
decision_threshold(
y,
p,
rule = c("youden", "kappa", "prevalence", "mpa"),
perc = 0.9,
...
)
# S3 method for class 'timesift'
decision_threshold(
y,
candidate = "ensemble",
rule = c("youden", "kappa", "prevalence", "mpa"),
perc = 0.9,
...
)
model_agreement(y, p_a, p_b, rule = c("youden", "kappa", "prevalence", "mpa"))Arguments
- y
Observed presence-absence,
0/1or logical; fordecision_threshold(), also atimesift()fit, which carries its own response and held-out predictions.- p
Predicted scores for the same units, in the same order. Higher means presence.
- rule
Threshold rule:
"youden","kappa","prevalence"or"mpa".kappa_score()cuts at"prevalence"by default, the rule the registeredkappametric reads, and the other functions at"youden".- ...
Ignored.
- perc
For
rule = "mpa", the share of presences the cut must keep,0.9by default. Where a rule is named without a call todecision_threshold()it takes that default.- candidate
For a
timesift()fit, the candidate whose cut is learned:"ensemble","selected"or the name of one, as predict() takes it.- p_a, p_b
Two models' predictions for the same units.
Value
For kappa_score(), one number. For decision_threshold(), the cut itself, applied as
p >= threshold, and on a fit one cut per response, named by it, NA where the response's
held-out predictions give none. For model_agreement(), a one-row data frame carrying the
agreement kappa between two models cut by the same rule, the share of units they decide
differently, and how often each is the one that is right there.
Details
Kappa is read at a threshold rather than maximised over one, so the rule that picks the
threshold is part of the statistic. "youden" is the operating point tss() is defined at and
inherits its selection bias; "kappa" maximises kappa itself and inherits the analogous bias;
"prevalence" cuts at the observed presence rate, which selects nothing from the labels and is
the rule to read an absolute level at. "mpa" is the minimum predicted area rule: the highest cut
that still predicts presence at a share perc of the observed presences, so the labels enter
through the presences alone.
Given a timesift() fit, decision_threshold() learns one cut per response from a candidate's
out-of-fold predictions of the fit's own targets, which is the cut predict(type = "binary")
applies to new targets.