Skip to contents

The metrics, the paired contrast between two arms on matched cells, every grain against a learner’s best, the inflation of a score read at its own best threshold, and what a fitted model read.

All of the Python reference

Installation

# Install from CRAN
install.packages("timesift")

# Or install the development version from GitHub
# install.packages("pak")
pak::pak("gcol33/timesift")
# Install from PyPI
pip install timesift

# with the torch encoders, the contrasts and the plots
pip install "timesift[torch,contrasts,plot]"

# Or install the development version from GitHub
pip install git+https://github.com/gcol33/timesift

tss()

tss(y, p, threshold=None)

Sensitivity plus specificity minus one, at the threshold that maximises it.

Given a threshold learned elsewhere, the score is read at that cut instead, presence being predicted at p >= threshold.

roc_auc()

roc_auc(y, p)

The area under the ROC curve, as the rank sum of the presences. Ties take the average rank.

average_precision()

average_precision(y, p)

The area under the precision-recall curve, as the step sum over the distinct predictions.

At each distinct prediction, the precision of calling every unit at or above it a presence, weighted by the share of presences that cut adds. Units sharing a prediction enter together. Its floor is the prevalence rather than one half, which makes it the reading of how well presences are ranked above absences where presences are rare.

kappa_score()

kappa_score(y, p, rule: str = 'prevalence')

Cohen’s kappa of a model’s decisions against the observed response, cut by rule.

The default cuts at the observed presence rate, the rule the registered kappa metric reads.

table_metric()

table_metric(y, p, metric: str, rule: str = 'youden', threshold=None, perc: float = 0.9)

A metric of the two-by-two table of decisions against observations.

metric is one of pod, pofd, far, sr, accuracy, bias, or, orss, csi and ets, biomod2’s evaluation statistics. The cut is the one rule of decision_threshold selects, or a threshold learned elsewhere, presence being predicted at p >= cut. A value the table does not define, a zero denominator, is NaN, as is a cell of one class.

boyce_index()

boyce_index(y, p, resolution: int = 100, width: float = 0.1)

The continuous Boyce index (Hirzel et al. 2006).

A window of width times the range of the predictions slides over that range in resolution + 1 equal steps. In each, the share of presences inside divided by the share of all units inside is the predicted-to-expected ratio, undefined where no unit falls inside, and the index is the Spearman correlation of that ratio with the window’s midpoint. The units of p are the background. The windows are closed at both ends and the ratio is not thinned of repeated values. NaN where the cell defines none, the predictions are all equal, the ratio is defined in fewer than three windows or takes one value.

regression_metric()

regression_metric(y, p, metric: str)

A metric of a numeric response and its predictions, as biomod2 reads an abundance model.

metric is r_squared (1 - sum(e**2) / sum((y - mean(y))**2), NaN where y is constant), pearson, rmse, mse, mae, max_error or poisson_deviance (mean(2 (y log(y / p) - (y - p))), the logarithm taken as zero at y = 0, NaN where a prediction is negative or is zero beside a count above zero). A comparison reads the highest score as the best, so the five errors are registered as neg_rmse, neg_mse, neg_mae, neg_max_error and neg_poisson_deviance with their sign reversed.

ordinal_metric()

ordinal_metric(y, p, metric: str)

A metric of ordinal classes, as biomod2 reads a model of an ordinal response.

The response is a column of whole-number classes, the model predicts a number on the same scale, and each prediction is read as the observed class nearest to it, the lower class on a tie. With m[i, j] the units of observed class j predicted as class i, over the k classes observed in the cell: accuracy is trace(m) / sum(m), recall the mean over the k classes of m[j, j] over the units of class j, precision the mean of m[i, i] over the units predicted as i, and f1 is 2 P R / (P + R) of the two means. A class with no unit, or in which nothing is predicted, adds zero to its mean. They are registered as ordinal_accuracy, ordinal_recall, ordinal_precision and ordinal_f1.

cohen_kappa()

cohen_kappa(a, b)

Chance-corrected agreement of two labellings of the same units, in either order.

decision_threshold()

decision_threshold(
    y,
    p=None,
    rule: str = 'youden',
    *,
    candidate: str = 'ensemble',
    perc: float = 0.9,
)

The probability cut a rule selects. Presence is predicted at p >= threshold.

"mpa" is the minimum predicted area rule: the highest cut that still predicts presence at a share perc of the observed presences.

Given a timesift() fit in place of y and no p, one cut per response, learned from candidate’s out-of-fold predictions of the fit’s own targets, as a dict keyed by the response; that is the cut predict(type="binary") applies to new targets.

model_agreement()

model_agreement(y, p_a, p_b, rule: str = 'youden')

Agreement between two models’ decisions, with how often each is right where they differ.

score_predictions()

score_predictions(y, p, folds, cells=None, metric: str = 'roc_auc')

Score held-out predictions on the cells the mask allows.

The scoring every arm of a ladder and every candidate of a run goes through, reachable on its own for a prediction matrix that came from somewhere else: a combination of arms, a model fitted outside the package, predictions read back from a file. A cell is one response in one fold, and the mask is computed from the response and the fold map alone, never from a model, which is what keeps two arms comparable.

paired_contrast()

paired_contrast(ladder: Ladder, a: str, b: str, interval: str = 'variables')

The difference between two arms, taken inside each cell both scored.

Two arms scored on the same held-out units do not necessarily have the same set of defined cells, so a difference of two marginal means is not a difference between the arms. Pairing removes the variation between variables and the part of a threshold-selected metric’s bias that the design sets, but not the part that belongs to each arm: how far TSS read at its best cut is inflated depends on how an arm’s predictions are distributed as well as on the presence counts, so two arms of equal skill on the same cell can carry different biases. A TSS contrast is best read beside the same contrast under roc_auc, which has no cut to choose.

Each arm is named whole, grain|learner. A learner named alone would take its best grain, chosen on the scores the contrast is then read off, and pairing does not remove that choice; select_grain chooses a grain on inner folds instead and contrasts the selection through compare. The interval is Student’s t on one degree of freedom fewer than there are variables, and p_method says whether the signed-rank p-value is "exact" or the "normal" approximation, which it is when the per-variable differences hold a zero or a tie or number fifty or more.

interval="nested_cv" replaces that interval with one for the difference in the two arms’ risk on a new sample, by the nested cross-validation of Bates, Hastie and Tibshirani (2024) read on the difference of the two arms’ cell scores, which needs a ladder fitted with grain_ladder(interval="nested_cv").

grain_contrasts()

grain_contrasts(
    ladder: Ladder,
    learner: str | None = None,
    reference: str | None = None,
    adjust: str = 'mvt',
)

Compare every grain against the reference grain of one learner.

A mixed model on the learner’s per-cell scores, score ~ grain + (1 | variable) + (1 | fold), fitted by restricted maximum likelihood, and every grain compared against reference by Dunnett’s many-to-one procedure. The design is balanced across variables, folds and grains, so each grain is compared within a variable and within a fold, and the variation between variables cancels from the comparison.

learner is the only one in the ladder by default, and reference the learner’s best grain. Taking the best-observed grain as the reference favours it, so read this beside paired_contrast, which singles out no grain. adjust is "mvt", the adjustment R’s default reads, or "none".

Returns one row per grain other than the reference: learner, grain, reference, the difference from the reference diff, its interval lower and upper, and the adjusted p_value. Needs scipy.

tss_inflation()

tss_inflation(
    y: Response,
    folds,
    skill=(0.6, 0.7, 0.9),
    replicates: int = 200,
    seed: int = 1,
)

How much a threshold chosen on the scored units inflates the level it reports.

The true skill statistic (TSS) is overestimated when the threshold is chosen to maximise TSS on the same held-out observations used to report the score. The fewer presences there are in the held-out data, the more TSS tends to overestimate population skill. tss_inflation estimates the TSS that a given sampling design would report for a known population skill. implied_skill reverses this calculation: given a reported TSS, it estimates the population skill that would yield that score on average under the same sampling design and assumed prediction distribution. This simulation-based matching follows the principle of indirect inference (Gouriéroux et al., 1993).

Predictions are simulated for each skill value as normal scores of equal variance, at the cell sizes and presence counts of this design, and the TSS is read at its best threshold the way a ladder reports a level. The mean reading minus the skill is the inflation. It is an expectation: a level is optimistic on average, not a bound every reading sits above. Predictions distributed otherwise inflate by a different amount at the same skill, which is also why a paired difference is not free of it.

implied_skill()

implied_skill(
    y: Response,
    folds,
    observed,
    grid=None,
    replicates: int = 200,
    seed: int = 1,
)

What population skill a level actually read is consistent with.

tss_inflation maps a population skill to the level a design reports for it; this inverts that map, under the distribution of predictions tss_inflation plants; a model whose predictions are distributed otherwise is inflated by a different amount. It does not correct a difference between two arms, whose inflations need not be equal.

occlusion()

occlusion(x, *args, **kwargs)

What one candidate’s score loses when a bin, or a channel, is withheld from it.

Takes a timesift.fit.timesift run or a timesift.ladder.grain_ladder result, and the profile itself is one implementation either way. The models kept per fold are the ones read, so the profile is measured where the score was: on the units each model held out.

response_curve()

response_curve(
    fit,
    candidate: str = 'ensemble',
    predictor=None,
    with_=None,
    fixed: str = 'mean',
    n: int = 50,
    spread: bool = False,
)

Vary one predictor of a fitted candidate across the range it takes, the rest held fixed.

A predictor is a cell of the representation the candidate reads: one statistic in one bin. Given as the name of a channel it is that statistic moved together in every bin, which is the way a column named in static is moved; given as {"bin": ..., "channel": ...} it is one cell, and a bare bin name is enough where the representation has a single channel. The reference is one made-up unit whose every cell holds the fixed summary ("mean", "median", "min" or "max") of that cell over the targets; a cell that is the same for every target keeps its value. The curve is read off the model fitted on all targets.

For the ensemble every member is moved the same way, a member that does not carry the predictor is held at the reference, and the members’ predictions are combined by the stack; spread adds the members’ standard deviation and interval. with_ is a second predictor, and the prediction is then read over the grid of the two; it is with in R, which Python reserves.

ResponseCurve

ResponseCurve(
    value,
    variable,
    prediction,
    candidate,
    predictor,
    fixed,
    value_with,
    with_label,
    sd,
    lower,
    upper,
)

The prediction against the value of a predictor, or of two.

prediction is [value, response]; with a second predictor value and value_with run over the grid of the two, the first fastest, as R’s table does. sd, lower and upper are the ensemble members’ spread at every value, and None where it was not asked.

Attributes:

  • value - np.ndarray
  • variable - tuple[str, …]
  • prediction - np.ndarray
  • candidate - str
  • predictor - str
  • fixed - str
  • value_with - np.ndarray | None
  • with_label - str | None
  • sd - np.ndarray | None
  • lower - np.ndarray | None
  • upper - np.ndarray | None