Fit an Inductive Venn-Abers Calibrator

Turns the uncalibrated probabilities of an already fitted binary or multiclass classifier into calibrated probabilities with an Inductive Venn-Abers predictor (IVAP). It needs the true classes and the model's probabilities for a separate calibration set; it creates neither predictive models nor data splits. Choose it when one held-out calibration set is affordable; use cross_venn_abers when data are scarce and one model per fold can be retained, or calibrated_workflow to let the package fit the model and split the data.

Usage

venn_abers(truth, probability, precision = NULL, method = "ovo")

Arguments

truth: A non-empty logical or numeric vector containing only FALSE/0 and TRUE/1. TRUE/1 is the event class. For multiclass, a non-empty factor without missing values, with at least two non-empty levels. Levels fix class order; every level must be observed in the calibration set.
probability: A numeric vector of uncalibrated event probabilities, the same length as truth, with finite values in [0, 1]. For factor truth, a numeric matrix with one row per truth and complete, unique, non-empty column names matching the factor levels. Columns are reordered by name. Entries must be finite and in [0, 1]; row sums must be within 1e-8 of one and are normalized within that tolerance. No clipping, name repair, or dropped rows.
precision: NULL (default) disables rounding. Otherwise, one finite numeric whole number, including zero or negative values, passed to native R round() for calibration probabilities only. Logical, missing, nonfinite, fractional, and multiple values are errors.
method: Exactly "ovo" (default) or "ovr". Explicitly supplying this argument requires factor truth and a class matrix.

Value

An S3 object of class "venn_abers": a list holding the fitted calibration state, from which new probabilities are calibrated without refitting. For binary truth it records the sorted unique calibration probabilities with their lower and upper Venn-Abers values and the calibration and event counts; for factor truth, the class levels, the method and one binary calibrator per class pair (OvO) or class (OvR). These list elements are internal and not public API. Use the methods instead:

predict.venn_abers returns the lower and upper values with the regularized event probability (binary), class probabilities (multiclass), class decisions or binary subproblem details, with the loss chosen at prediction time.
plot.venn_abers draws the stored calibration map and returns it invisibly as a data frame.
print.venn_abers summarizes the calibration size, events and probability knots, or the method, calibration size, classes and number of subproblems.

No predictive model or new data are stored.

Details

The calibration set must be separate from the fixed predictive model's training, including learned preprocessing and tuning. Single-class and singleton binary calibration sets are valid. Rows align by position, not names.

One-vs-One (method = "ovo") uses each pair in factor-level order and only that pair's calibration observations. The later class is the binary event. Its probability is divided by the pair sum before optional rounding. An exactly zero pair sum uses event probability 0.5, including during prediction. This fit replacement is also rounded: at precision = 0, native R rounds 0.5 to zero. Only the event probability is rounded, not its complement independently. There are no separate trained pair models. All global classes must occur, so an Inductive OvO pair always has both observed classes. The underlying binary calibrator still accepts single-class subsets.

One-vs-Rest (method = "ovr") uses all calibration observations for each class. That class is the binary event; its class probability is the calibration probability, optionally rounded at fit time. The binary points are normalized across classes only after the chosen regularization.

Original probabilities are validated before rounding; an out-of-range input cannot become valid through rounding. Exact ties after rounding are combined. New probabilities supplied to predict() remain unrounded, and changing precision requires a new fit. Native R rounding can differ from NumPy at floating-point boundaries; NumPy rounding is not emulated.

Choose log-loss (default) or Brier regularization at prediction time without refitting. See predict.venn_abers for probabilities, classes, pair coupling or class normalization, and details. Manual Brier for both multiclass methods is a deliberate R extension over the examined Python 1.5.4 manual path, which supports log regularization and precision. Exactly two factor levels also select the multiclass matrix contract, with its distinct tie rule and Brier score scale.

Examples

calibrator <- venn_abers(
  truth = c(FALSE, TRUE, FALSE, TRUE),
  probability = c(0.1, 0.4, 0.35, 0.8)
)
calibrator
#> <venn_abers>
#> Calibration observations: 4
#> Event observations: 2
#> Unique probability knots: 4

predict(calibrator, c(0.2, 0.5, 0.9))
#>       lower upper probability
#> 1 0.0000000   0.5   0.3333333
#> 2 0.5000000   1.0   0.6666667
#> 3 0.6666667   1.0   0.7500000
rounded <- venn_abers(c(0, 1, 0, 1), c(.501, .502, .509, .510), precision = 2)
predict(rounded, c(.501, .505, .51), loss = "brier")
#>       lower     upper probability
#> 1 0.3333333 0.6666667   0.5000000
#> 2 0.3333333 0.6666667   0.5000000
#> 3 0.4000000 0.6666667   0.5244444

# Fixed-model probabilities on a separate calibration set.
classes <- c("cat", "dog", "bird")
truth <- factor(rep(classes, 2), levels = classes)
p <- rbind(c(.7, .2, .1), c(.2, .6, .2), c(.1, .2, .7),
           c(.5, .3, .2), c(.3, .5, .2), c(.2, .3, .5))
colnames(p) <- classes
fit <- venn_abers(truth, p, precision = 2)
new_p <- rbind(c(.6, .3, .1), c(.1, .4, .5))
colnames(new_p) <- classes
assessment <- factor(c("cat", "bird"), levels = classes)
details <- predict(fit, new_p, loss = "brier", type = "details")
details$pairs
#>   row non_event event     lower     upper probability
#> 1   1       cat   dog 0.0000000 0.5000000   0.3750000
#> 2   2       cat   dog 0.6666667 1.0000000   0.7222222
#> 3   1       cat  bird 0.0000000 0.5000000   0.3750000
#> 4   2       cat  bird 0.5000000 1.0000000   0.6250000
#> 5   1       dog  bird 0.0000000 0.3333333   0.2777778
#> 6   2       dog  bird 0.0000000 1.0000000   0.5000000
predict(fit, new_p, loss = "brier", type = "class")
#> [1] cat dog
#> Levels: cat dog bird
brier_score(assessment, details$prediction)
#> [1] 0.5127985
log_loss(assessment, details$prediction)
#> [1] 0.8547741
calibration_plot(assessment, details$prediction)


ovr <- venn_abers(truth, p, method = "ovr", precision = 2)
predict(ovr, new_p)
#>            cat       dog      bird
#> [1,] 0.4878049 0.3658537 0.1463415
#> [2,] 0.1250000 0.3750000 0.5000000
ovr_details <- predict(ovr, new_p, loss = "brier", type = "details")
ovr_details$classes
#>   row event lower upper probability
#> 1   1   cat   0.5  1.00     0.62500
#> 2   2   cat   0.0  0.20     0.18000
#> 3   1   dog   0.0  1.00     0.50000
#> 4   2   dog   0.0  1.00     0.50000
#> 5   1  bird   0.0  0.25     0.21875
#> 6   2  bird   0.5  1.00     0.62500
predict(ovr, new_p, loss = "brier", type = "class")
#> [1] cat  bird
#> Levels: cat dog bird
brier_score(assessment, ovr_details$prediction)
#> [1] 0.444197
log_loss(assessment, ovr_details$prediction)
#> [1] 0.7508373
calibration_plot(assessment, ovr_details$prediction)


# Installed standalone examples for binary options (precision, loss, event
# selection, Cross) and both multiclass methods:
system.file("examples", c("binary-options.R", "inductive-ovo.R", "inductive-ovr.R"),
  package = "vennabers")
#> [1] "/tmp/RtmpBnTCfh/temp_libpath192478f23885/vennabers/examples/binary-options.R"
#> [2] "/tmp/RtmpBnTCfh/temp_libpath192478f23885/vennabers/examples/inductive-ovo.R" 
#> [3] "/tmp/RtmpBnTCfh/temp_libpath192478f23885/vennabers/examples/inductive-ovr.R"

Usage

Arguments

Value

Details

See also

Examples