Held-out threshold at a fixed sensitivity

The threshold of the range rule's first test. Pooled over the cross-validation folds, it is the value that keeps sens of the presences the fold's model never saw (sens_threshold() on the held-out predictions). Published thresholds are fitted on training data, where a flexible model scores its own presences too high; this one is not.

Usage

heldout_threshold(cv, cv_train = NULL, sens = 0.9)

Arguments

cv

data frame of held-out predictions: fold, presence (1/0), pred

cv_train

optional data frame of the same shape: each fold's model on its training rows

sens

sensitivity to fix

Value

a list: thr (the pooled held-out threshold), pooled (confusion_at() of all held-out rows at thr) and folds (one confusion_at() row per fold, with fold and thr_from = "train" or "pooled")

Details

When the predictions of each fold's model on its own training rows are supplied (cv_train), the per-fold table is the honest confusion matrix: the threshold is fixed at sens on the fold's TRAINING presences and the counts are taken on its TEST rows. Its sens column then says how much sensitivity a training threshold really delivers.