Fit a Binary or Multiclass Cross Venn-Abers Calibrator
cross_venn_abers.RdCalibrates binary or multiclass classifier probabilities with a Cross
Venn-Abers predictor (CVAP): every fold calibrates the model that was trained
without it, and prediction averages the fold results. It needs the true
classes, the out-of-fold probabilities of one retained model per fold and the
fold labels; it fits no models itself. Choose it over
venn_abers when a single held-out calibration set would cost too
much training data, or use calibrated_workflow with
method = "cross" to let the package fit the fold models.
Arguments
- truth
A non-empty logical or numeric/integer vector containing only
FALSE/0andTRUE/1. The event class is alwaysTRUE/1. Alternatively, a factor with at least two levels for multiclass calibration; every level must be globally observed.- probability
A numeric/integer vector of out-of-fold event probabilities, finite and in
[0, 1], aligned withtruth. For factor truth, a numeric matrix with one row per observation and complete, unique class column names matching its levels; seevenn_abers.- fold
An aligned vector of non-empty, non-missing character labels or finite positive whole-number numeric/integer IDs. Character matching is literal, without trimming or case folding. Numeric IDs match their Base R
as.characterrepresentations. Repeated IDs indicate membership in the same fold. Factors and logical IDs are not supported.- precision
NULL(default) disables rounding. Otherwise, one finite numeric whole number, including zero or negative values, used for native R rounding of calibration probabilities in every fold. Seevenn_abersfor validation and rounding semantics.- method
For factor truth, exactly
"ovo"(default) or"ovr". Explicit use with binary vector truth is an error.
Value
An S3 object of class "cross_venn_abers": a list retaining fold
identities and calibration state for repeated predictions from the matching
fold models. For binary truth it records the fold IDs in first-observed order,
the total calibration size and one Inductive venn_abers calibrator per
fold; for factor truth, the class levels, the method, the fold IDs and one
Cross calibrator per class pair (OvO) or class (OvR). These list elements are
internal and not public API. Use the methods instead:
predict.cross_venn_aberscombines the fold calibrators on new probabilities from each fold model into regularized probabilities, class decisions or per-fold details, with the loss chosen at prediction time.plot.cross_venn_abersdraws each fold's stored calibration map and returns the maps invisibly as a data frame.print.cross_venn_aberssummarizes the number of folds and calibration observations, and the classes for multiclass.
Predictive models are retained separately by the caller.
Details
The function fits Inductive Venn-Abers calibrators per calibration fold and, for multiclass, per class pair or class against the rest. Prediction combines their results with equal-weight log-loss (default) or Brier aggregation to produce a regularized event probability.
For binary calibration, all three inputs must be vectors of equal, nonzero length with no missing values. Names attached to vector elements are not alignment keys: positions must refer to the same observation. Invalid types, shapes, nonbinary truth, nonfinite values, or out-of-range probabilities cause an error for the entire call. No rows are dropped and probabilities are not clamped or coerced from other representations. Original probabilities are checked before optional rounding; exact ties after rounding are combined. New prediction values are never rounded. Changing precision requires refitting, while loss is chosen at prediction time. R and NumPy rounding can differ at floating-point boundaries.
Supply one non-overlapping partition with at least two represented non-empty
folds. Unequal sizes, singleton folds, and single-class folds are supported;
each fold receives equal aggregation weight regardless of its size. There is
no separate empty-fold declaration. Use venn_abers() for one calibration
set.
Each fold model must exclude its calibration fold from the entire learned training pipeline, including preprocessing and tuning. That same model must produce both its fold's out-of-fold probabilities and its corresponding named column of new-observation probabilities. Retain those models for subsequent prediction: a single model refitted on all data is not a replacement. Keep evaluation truth outside model fitting and calibration.
Callers ensure each underlying calibration observation appears once and that new-observation rows align across model columns. The supplied arrays cannot prove model provenance, training exclusion, absence of leakage, unique underlying observations, or semantic row alignment.
Cross output defaults to a regularized probability. Classes and individual
fold results are available via predict.cross_venn_abers. There is
no aggregate interval. Geometric aggregate endpoints can cross.
Neither those endpoints nor the combined point inherit the IVAP
perfect-calibration guarantee. They are not confidence intervals, and score
improvement is not guaranteed on every data set.
Unsupported inputs and features
Package-managed predictive-model fitting, fold generation, preprocessing, tuning or resampling; repeated or overlapping partitions; regression calibration; separate pairwise model training or pair-specific fold plans; observation weights; alternate event-class selection; automatic coercion or missing-value handling; fold-size weighting or pooling; public aggregate endpoints; formula or data-frame cross interfaces; tidymodels/yardstick integrations; and new metrics, smoothing or plotting features are not supported. Only Base R and standard R distribution packages are needed. Installation and acceptance require neither Python nor networking; Python reference-fixture regeneration is optional and separate.
Multiclass with a shared fold plan
Factor levels fix the global class order. Supply one out-of-fold class matrix and a shared fold vector. All classes must be observed globally; individual folds may omit classes. Each OvO pair must have at least one observation in every fold. Empty pairs fail with their class names and fold ID; singleton and single-class binary subcalibrations are allowed. OvR uses every row in each fold. No pair-specific training plans are introduced.
Class matrices must have finite entries in [0, 1] and row sums within
1e-8 of one; accepted rows are normalized. In OvO, pair normalization
precedes calibration rounding, with zero pair sums replaced by an event
probability of 0.5 (also subject to fit rounding). The later class in factor
order is the event; in OvR, the selected class is the event against the rest.
New prediction probabilities remain unrounded.
Predict with a list of class matrices named by fold ID, from the corresponding retained models, for the same new observations in the same order. Binary Cross aggregation happens separately for each subproblem before reciprocal OvO coupling or OvR normalization. Log loss uses the stable equal-weight endpoint merger; Brier uses the equal-weight mean of binary fold points. No pooling, size weighting, or averaging of finished multiclass fold outputs is performed. Scores and plots use the final class matrix.
This manual shared-fold interface is an R extension. Python 1.5.4 has no corresponding public manual Cross-multiclass whole-procedure API; numerical checks use binary reference components and independent combination. Binary diagnostics are not multiclass or aggregate Cross confidence intervals.
See also
vignette("method", package = "vennabers") for the theory and
its guarantees, vignette("manual-calibration", package = "vennabers")
for a worked introduction, and vennabers_examples for annotated
installed scripts.
predict.cross_venn_abers,
print.cross_venn_abers, brier_score,
log_loss, calibration_plot
Examples
# Precomputed probabilities from three retained fold models. Each model
# excluded its corresponding calibration fold from its learned pipeline.
calibrator <- cross_venn_abers(
truth = c(FALSE, TRUE, FALSE, TRUE, FALSE, TRUE),
probability = c(.1, .8, .2, .9, .3, .7),
fold = c("A", "A", "B", "B", "C", "C")
)
print(calibrator)
#> <cross_venn_abers>
#> Folds: 3
#> Calibration observations: 6
# Each row is the same new observation across all corresponding fold models.
newdata <- cbind(A = c(.2, .8), B = c(.3, .7), C = c(.1, .9))
result <- predict(calibrator, newdata)
result
#> probability
#> 1 0.4424933
#> 2 0.6135118
test_truth <- c(FALSE, TRUE) # Untouched by fitting and calibration.
brier_score(test_truth, result$probability)
#> [1] 0.1725867
log_loss(test_truth, result$probability)
#> [1] 0.5364183
calibration_plot(test_truth, result$probability)
# After installation, find the standalone example with:
# system.file("examples", "cross-multiclass.R", package = "vennabers")
library(vennabers)
# Precomputed out-of-fold probabilities from two retained multiclass models.
# Model A excluded rows 1:2; model B excluded rows 3:6, including preprocessing
# and tuning. Retain both models yourself. No model is trained by this package.
classes <- c("cat", "dog", "bird")
truth <- factor(c("cat", "dog", "cat", "bird", "dog", "bird"), levels = classes)
fold <- c("A", "A", "B", "B", "B", "B")
p <- rbind(c(.7, .2, .1), c(.2, .6, .2), c(.5, .3, .2),
c(.1, .2, .7), c(.3, .5, .2), c(.2, .3, .5))
colnames(p) <- classes
# The same three untouched assessment observations, in the same row order,
# predicted by each corresponding retained model. List order is immaterial.
new_p <- list(B = rbind(c(.6, .3, .1), c(.1, .4, .5), c(.1, .8, .1)),
A = rbind(c(.5, .4, .1), c(.2, .2, .6), c(.2, .7, .1)))
new_p <- lapply(new_p, function(x) { colnames(x) <- classes; x })
assessment <- factor(c("cat", "bird", "dog"), levels = classes)
local({
plot_file <- tempfile(fileext = ".pdf")
grDevices::pdf(plot_file)
on.exit({ grDevices::dev.off(); unlink(plot_file) })
for (method in c("ovo", "ovr")) {
fit <- cross_venn_abers(truth, p, fold, method = method, precision = 2)
print(fit)
for (loss in c("log", "brier")) {
details <- predict(fit, new_p, loss = loss, type = "details")
probability <- predict(fit, new_p, loss = loss)
predicted_class <- predict(fit, new_p, loss = loss, type = "class")
# Reconstruct from the fold endpoints/points, then combine subproblems.
parts <- details[[2L]]
problems <- unique(parts[c(if (method == "ovo") "non_event", "event")])
merged <- vapply(seq_len(nrow(problems)), function(i) {
selected <- parts$event == problems$event[i]
if (method == "ovo") selected <- selected & parts$non_event == problems$non_event[i]
vapply(seq_len(nrow(probability)), function(row) {
f <- parts[selected & parts$row == row, ]
if (loss == "log") {
plogis(mean(log(f$upper)) - mean(log1p(-f$lower)))
} else mean(f$probability)
}, 0)
}, numeric(nrow(probability)))
recovered <- if (method == "ovr") merged else vapply(classes, function(class) {
q <- cbind(1 - merged[, problems$non_event == class, drop = FALSE],
merged[, problems$event == class, drop = FALSE])
1 / (1 + rowSums((1 - q) / q))
}, numeric(nrow(probability)))
recovered <- recovered / rowSums(recovered)
stopifnot(max(abs(recovered - probability)) <= 1e-10,
identical(probability, details$prediction),
identical(predicted_class,
factor(classes[max.col(probability, ties.method = "first")],
levels = classes)))
print(details)
print(predicted_class)
print(c(brier = brier_score(assessment, probability),
log = log_loss(assessment, probability)))
groups <- calibration_plot(assessment, probability, bins = 1)
stopifnot(max(abs(groups$mean_probability - colMeans(probability))) <= 1e-10,
all(groups$count == length(assessment)), all(groups$event_rate == 1 / 3))
print(groups)
}
}
})
#> <cross_venn_abers: ovo>
#> Classes: cat, dog, bird
#> Folds: 2
#> Calibration observations: 6
#> $prediction
#> cat dog bird
#> [1,] 0.4209914 0.3487607 0.2302479
#> [2,] 0.2612039 0.3693981 0.3693981
#> [3,] 0.2500000 0.5000000 0.2500000
#>
#> $pairs
#> row non_event event fold lower upper probability
#> 1 1 cat dog A 0.0 1.0 0.5000000
#> 2 2 cat dog A 0.0 1.0 0.5000000
#> 3 3 cat dog A 0.5 1.0 0.6666667
#> 4 1 cat dog B 0.0 0.5 0.3333333
#> 5 2 cat dog B 0.5 1.0 0.6666667
#> 6 3 cat dog B 0.5 1.0 0.6666667
#> 7 1 cat bird A 0.0 1.0 0.5000000
#> 8 2 cat bird A 0.0 1.0 0.5000000
#> 9 3 cat bird A 0.0 1.0 0.5000000
#> 10 1 cat bird B 0.0 0.5 0.3333333
#> 11 2 cat bird B 0.5 1.0 0.6666667
#> 12 3 cat bird B 0.0 1.0 0.5000000
#> 13 1 dog bird A 0.0 0.5 0.3333333
#> 14 2 dog bird A 0.0 1.0 0.5000000
#> 15 3 dog bird A 0.0 0.5 0.3333333
#> 16 1 dog bird B 0.0 0.5 0.3333333
#> 17 2 dog bird B 0.0 1.0 0.5000000
#> 18 3 dog bird B 0.0 0.5 0.3333333
#>
#> [1] cat dog dog
#> Levels: cat dog bird
#> brier log
#> 0.4957467 0.8513901
#> class mean_probability event_rate count
#> 1 cat 0.3107318 0.3333333 3
#> 2 dog 0.4060529 0.3333333 3
#> 3 bird 0.2832153 0.3333333 3
#> $prediction
#> cat dog bird
#> [1,] 0.3949174 0.3497344 0.2553482
#> [2,] 0.2800000 0.3600000 0.3600000
#> [3,] 0.2727273 0.4545455 0.2727273
#>
#> $pairs
#> row non_event event fold lower upper probability
#> 1 1 cat dog A 0.0 1.0 0.500
#> 2 2 cat dog A 0.0 1.0 0.500
#> 3 3 cat dog A 0.5 1.0 0.625
#> 4 1 cat dog B 0.0 0.5 0.375
#> 5 2 cat dog B 0.5 1.0 0.625
#> 6 3 cat dog B 0.5 1.0 0.625
#> 7 1 cat bird A 0.0 1.0 0.500
#> 8 2 cat bird A 0.0 1.0 0.500
#> 9 3 cat bird A 0.0 1.0 0.500
#> 10 1 cat bird B 0.0 0.5 0.375
#> 11 2 cat bird B 0.5 1.0 0.625
#> 12 3 cat bird B 0.0 1.0 0.500
#> 13 1 dog bird A 0.0 0.5 0.375
#> 14 2 dog bird A 0.0 1.0 0.500
#> 15 3 dog bird A 0.0 0.5 0.375
#> 16 1 dog bird B 0.0 0.5 0.375
#> 17 2 dog bird B 0.0 1.0 0.500
#> 18 3 dog bird B 0.0 0.5 0.375
#>
#> [1] cat dog dog
#> Levels: cat dog bird
#> brier log
#> 0.5391742 0.9130624
#> class mean_probability event_rate count
#> 1 cat 0.3158816 0.3333333 3
#> 2 dog 0.3880933 0.3333333 3
#> 3 bird 0.2960252 0.3333333 3
#> <cross_venn_abers: ovr>
#> Classes: cat, dog, bird
#> Folds: 2
#> Calibration observations: 6
#> $prediction
#> cat dog bird
#> [1,] 0.4385330 0.3743113 0.1871557
#> [2,] 0.2071068 0.3284271 0.4644661
#> [3,] 0.2584184 0.5168368 0.2247449
#>
#> $classes
#> row event fold lower upper probability
#> 1 1 cat A 0.0 1.0000000 0.5000000
#> 2 2 cat A 0.0 0.5000000 0.3333333
#> 3 3 cat A 0.0 1.0000000 0.5000000
#> 4 1 cat B 0.5 1.0000000 0.6666667
#> 5 2 cat B 0.0 0.2500000 0.2000000
#> 6 3 cat B 0.0 0.2500000 0.2000000
#> 7 1 dog A 0.0 1.0000000 0.5000000
#> 8 2 dog A 0.0 0.5000000 0.3333333
#> 9 3 dog A 0.5 1.0000000 0.6666667
#> 10 1 dog B 0.0 1.0000000 0.5000000
#> 11 2 dog B 0.0 1.0000000 0.5000000
#> 12 3 dog B 0.5 1.0000000 0.6666667
#> 13 1 bird A 0.0 0.3333333 0.2500000
#> 14 2 bird A 0.0 1.0000000 0.5000000
#> 15 3 bird A 0.0 0.5000000 0.3333333
#> 16 1 bird B 0.0 0.3333333 0.2500000
#> 17 2 bird B 0.5 1.0000000 0.6666667
#> 18 3 bird B 0.0 0.3333333 0.2500000
#>
#> [1] cat bird dog
#> Levels: cat dog bird
#> brier log
#> 0.4262242 0.7504050
#> class mean_probability event_rate count
#> 1 cat 0.3013527 0.3333333 3
#> 2 dog 0.4065251 0.3333333 3
#> 3 bird 0.2921222 0.3333333 3
#> $prediction
#> cat dog bird
#> [1,] 0.4196891 0.3730570 0.2072539
#> [2,] 0.2289157 0.3373494 0.4337349
#> [3,] 0.2741722 0.4768212 0.2490066
#>
#> $classes
#> row event fold lower upper probability
#> 1 1 cat A 0.0 1.0000000 0.5000000
#> 2 2 cat A 0.0 0.5000000 0.3750000
#> 3 3 cat A 0.0 1.0000000 0.5000000
#> 4 1 cat B 0.5 1.0000000 0.6250000
#> 5 2 cat B 0.0 0.2500000 0.2187500
#> 6 3 cat B 0.0 0.2500000 0.2187500
#> 7 1 dog A 0.0 1.0000000 0.5000000
#> 8 2 dog A 0.0 0.5000000 0.3750000
#> 9 3 dog A 0.5 1.0000000 0.6250000
#> 10 1 dog B 0.0 1.0000000 0.5000000
#> 11 2 dog B 0.0 1.0000000 0.5000000
#> 12 3 dog B 0.5 1.0000000 0.6250000
#> 13 1 bird A 0.0 0.3333333 0.2777778
#> 14 2 bird A 0.0 1.0000000 0.5000000
#> 15 3 bird A 0.0 0.5000000 0.3750000
#> 16 1 bird B 0.0 0.3333333 0.2777778
#> 17 2 bird B 0.5 1.0000000 0.6250000
#> 18 3 bird B 0.0 0.3333333 0.2777778
#>
#> [1] cat bird dog
#> Levels: cat dog bird
#> brier log
#> 0.4722134 0.8147255
#> class mean_probability event_rate count
#> 1 cat 0.3075923 0.3333333 3
#> 2 dog 0.3957425 0.3333333 3
#> 3 bird 0.2966651 0.3333333 3
# One shared fold plan serves all pairs/classes. Fold A lacks bird but every
# OvO pair remains nonempty. Empty pairs fail with the pair and fold identified.
# Endpoints describe binary subproblems, with no aggregate confidence interval.
# Manual Cross multiclass is an R extension; Python has no matching public
# manual whole-procedure API. Numerical tests reconstruct binary components.