The metrics, the paired contrast between two arms on matched cells, every grain against a learner’s best, the inflation of a score read at its own best threshold, and what a fitted model read.
Installation
# Install from CRAN
install.packages("timesift")
# Or install the development version from GitHub
# install.packages("pak")
pak::pak("gcol33/timesift")
tss()
Sensitivity plus specificity minus one, at the threshold that maximises it.
Given a threshold learned elsewhere, the score is read
at that cut instead, presence being predicted at
p >= threshold.
roc_auc()
The area under the ROC curve, as the rank sum of the presences. Ties take the average rank.
average_precision()
The area under the precision-recall curve, as the step sum over the distinct predictions.
At each distinct prediction, the precision of calling every unit at or above it a presence, weighted by the share of presences that cut adds. Units sharing a prediction enter together. Its floor is the prevalence rather than one half, which makes it the reading of how well presences are ranked above absences where presences are rare.
kappa_score()
Cohen’s kappa of a model’s decisions against the observed response,
cut by rule.
The default cuts at the observed presence rate, the rule the
registered kappa metric reads.
table_metric()
A metric of the two-by-two table of decisions against observations.
metric is one of pod, pofd,
far, sr, accuracy,
bias, or, orss, csi
and ets, biomod2’s evaluation statistics. The cut is the
one rule of decision_threshold selects, or a
threshold learned elsewhere, presence being predicted at
p >= cut. A value the table does not define, a zero
denominator, is NaN, as is a cell of one class.
boyce_index()
The continuous Boyce index (Hirzel et al. 2006).
A window of width times the range of the predictions
slides over that range in resolution + 1 equal steps. In
each, the share of presences inside divided by the share of all units
inside is the predicted-to-expected ratio, undefined where no unit falls
inside, and the index is the Spearman correlation of that ratio with the
window’s midpoint. The units of p are the background. The
windows are closed at both ends and the ratio is not thinned of repeated
values. NaN where the cell defines none, the predictions are all equal,
the ratio is defined in fewer than three windows or takes one value.
regression_metric()
A metric of a numeric response and its predictions, as biomod2 reads an abundance model.
metric is r_squared
(1 - sum(e**2) / sum((y - mean(y))**2), NaN where
y is constant), pearson, rmse,
mse, mae, max_error or
poisson_deviance
(mean(2 (y log(y / p) - (y - p))), the logarithm taken as
zero at y = 0, NaN where a prediction is negative or is
zero beside a count above zero). A comparison reads the highest score as
the best, so the five errors are registered as neg_rmse,
neg_mse, neg_mae, neg_max_error
and neg_poisson_deviance with their sign reversed.
ordinal_metric()
A metric of ordinal classes, as biomod2 reads a model of an ordinal response.
The response is a column of whole-number classes, the model predicts
a number on the same scale, and each prediction is read as the observed
class nearest to it, the lower class on a tie. With m[i, j]
the units of observed class j predicted as class
i, over the k classes observed in the cell:
accuracy is trace(m) / sum(m),
recall the mean over the k classes of
m[j, j] over the units of class j,
precision the mean of m[i, i] over the units
predicted as i, and f1 is
2 P R / (P + R) of the two means. A class with no unit, or
in which nothing is predicted, adds zero to its mean. They are
registered as ordinal_accuracy,
ordinal_recall, ordinal_precision and
ordinal_f1.
decision_threshold()
decision_threshold(
y,
p=None,
rule: str = 'youden',
*,
candidate: str = 'ensemble',
perc: float = 0.9,
)The probability cut a rule selects. Presence is predicted at
p >= threshold.
"mpa" is the minimum predicted area rule: the highest
cut that still predicts presence at a share perc of the
observed presences.
Given a timesift() fit in place of y and no
p, one cut per response, learned from
candidate’s out-of-fold predictions of the fit’s own
targets, as a dict keyed by the response; that is the cut
predict(type="binary") applies to new targets.
model_agreement()
Agreement between two models’ decisions, with how often each is right where they differ.
score_predictions()
Score held-out predictions on the cells the mask allows.
The scoring every arm of a ladder and every candidate of a run goes through, reachable on its own for a prediction matrix that came from somewhere else: a combination of arms, a model fitted outside the package, predictions read back from a file. A cell is one response in one fold, and the mask is computed from the response and the fold map alone, never from a model, which is what keeps two arms comparable.
paired_contrast()
The difference between two arms, taken inside each cell both scored.
Two arms scored on the same held-out units do not necessarily have
the same set of defined cells, so a difference of two marginal means is
not a difference between the arms. Pairing removes the variation between
variables and the part of a threshold-selected metric’s bias that the
design sets, but not the part that belongs to each arm: how far TSS read
at its best cut is inflated depends on how an arm’s predictions are
distributed as well as on the presence counts, so two arms of equal
skill on the same cell can carry different biases. A TSS contrast is
best read beside the same contrast under roc_auc, which has
no cut to choose.
Each arm is named whole, grain|learner. A learner named
alone would take its best grain, chosen on the scores the contrast is
then read off, and pairing does not remove that choice;
select_grain chooses a grain on inner folds instead and
contrasts the selection through compare. The interval is
Student’s t on one degree of freedom fewer than there are variables, and
p_method says whether the signed-rank p-value is
"exact" or the "normal" approximation, which
it is when the per-variable differences hold a zero or a tie or number
fifty or more.
interval="nested_cv" replaces that interval with one for
the difference in the two arms’ risk on a new sample, by the nested
cross-validation of Bates, Hastie and Tibshirani (2024) read on the
difference of the two arms’ cell scores, which needs a ladder fitted
with grain_ladder(interval="nested_cv").
grain_contrasts()
grain_contrasts(
ladder: Ladder,
learner: str | None = None,
reference: str | None = None,
adjust: str = 'mvt',
)Compare every grain against the reference grain of one learner.
A mixed model on the learner’s per-cell scores,
score ~ grain + (1 | variable) + (1 | fold), fitted by
restricted maximum likelihood, and every grain compared against
reference by Dunnett’s many-to-one procedure. The design is
balanced across variables, folds and grains, so each grain is compared
within a variable and within a fold, and the variation between variables
cancels from the comparison.
learner is the only one in the ladder by default, and
reference the learner’s best grain. Taking the
best-observed grain as the reference favours it, so read this beside
paired_contrast, which singles out no grain.
adjust is "mvt", the adjustment R’s default
reads, or "none".
Returns one row per grain other than the reference:
learner, grain, reference, the
difference from the reference diff, its interval
lower and upper, and the adjusted
p_value. Needs scipy.
tss_inflation()
How much a threshold chosen on the scored units inflates the level it reports.
The true skill statistic (TSS) is overestimated when the threshold is
chosen to maximise TSS on the same held-out observations used to report
the score. The fewer presences there are in the held-out data, the more
TSS tends to overestimate population skill. tss_inflation
estimates the TSS that a given sampling design would report for a known
population skill. implied_skill reverses this calculation:
given a reported TSS, it estimates the population skill that would yield
that score on average under the same sampling design and assumed
prediction distribution. This simulation-based matching follows the
principle of indirect inference (Gouriéroux
et al., 1993).
Predictions are simulated for each skill value as normal scores of equal variance, at the cell sizes and presence counts of this design, and the TSS is read at its best threshold the way a ladder reports a level. The mean reading minus the skill is the inflation. It is an expectation: a level is optimistic on average, not a bound every reading sits above. Predictions distributed otherwise inflate by a different amount at the same skill, which is also why a paired difference is not free of it.
implied_skill()
What population skill a level actually read is consistent with.
tss_inflation maps a population skill to the level a
design reports for it; this inverts that map, under the distribution of
predictions tss_inflation plants; a model whose predictions
are distributed otherwise is inflated by a different amount. It does not
correct a difference between two arms, whose inflations need not be
equal.
occlusion()
What one candidate’s score loses when a bin, or a channel, is withheld from it.
Takes a timesift.fit.timesift run or a
timesift.ladder.grain_ladder result, and the profile itself
is one implementation either way. The models kept per fold are the ones
read, so the profile is measured where the score was: on the units each
model held out.
response_curve()
response_curve(
fit,
candidate: str = 'ensemble',
predictor=None,
with_=None,
fixed: str = 'mean',
n: int = 50,
spread: bool = False,
)Vary one predictor of a fitted candidate across the range it takes, the rest held fixed.
A predictor is a cell of the representation the candidate reads: one
statistic in one bin. Given as the name of a channel it is that
statistic moved together in every bin, which is the way a column named
in static is moved; given as
{"bin": ..., "channel": ...} it is one cell, and a bare bin
name is enough where the representation has a single channel. The
reference is one made-up unit whose every cell holds the
fixed summary ("mean", "median",
"min" or "max") of that cell over the targets;
a cell that is the same for every target keeps its value. The curve is
read off the model fitted on all targets.
For the ensemble every member is moved the same way, a member that
does not carry the predictor is held at the reference, and the members’
predictions are combined by the stack; spread adds the
members’ standard deviation and interval. with_ is a second
predictor, and the prediction is then read over the grid of the two; it
is with in R, which Python reserves.
ResponseCurve
ResponseCurve(
value,
variable,
prediction,
candidate,
predictor,
fixed,
value_with,
with_label,
sd,
lower,
upper,
)The prediction against the value of a predictor, or of two.
prediction is [value, response]; with a
second predictor value and value_with run over
the grid of the two, the first fastest, as R’s table does.
sd, lower and upper are the
ensemble members’ spread at every value, and None where it
was not asked.
Attributes:
-
value- np.ndarray -
variable- tuple[str, …] -
prediction- np.ndarray -
candidate- str -
predictor- str -
fixed- str -
value_with- np.ndarray | None -
with_label- str | None -
sd- np.ndarray | None -
lower- np.ndarray | None -
upper- np.ndarray | None