Venn-Abers Classification and Regression Calibration

Calibrate binary and multiclass classification probabilities or real-valued regression predictions using Inductive or Cross Venn-Abers methods. Supply predictions from an existing model, or use optional tidymodels workflows to train preprocessing, models and calibrators together.

Start with vignette("method", package = "vennabers"): it explains what Venn-Abers predictors compute, which guarantees the literature gives and which it does not, and checks the formulas against the package.

Calibrating supplied predictions

The manual interfaces require no additional R packages. Use venn_abers for Inductive classification calibration and cross_venn_abers for matched out-of-fold probabilities. Both support binary inputs and multiclass one-vs-one or one-vs-rest calibration. For numeric predictions, use venn_abers_regression or cross_venn_abers_regression. venn_abers_one_hot represents class decisions as a 0/1 matrix with one column per class.

Continue with vignette("manual-calibration", package = "vennabers").

Training and calibrating workflows

Use calibrated_workflow with an unfitted tidymodels workflow, or calibrate_workflow to calibrate an already fitted workflow on new data. These interfaces require the optional packages in DESCRIPTION. model_adapter turns your own fit and predict functions into a workflow model without registering a parsnip engine. Follow vignette("workflows", package = "vennabers") for a complete example, then vignette("tuning", package = "vennabers") for Grid/Bayes selection and independent assessment.

workflow_training documents splits and prediction models; workflow_postprocessing covers learned postprocessors. Use venn_abers_details for prediction diagnostics and venn_abers_tuning for search configuration and results.

Example scripts

vennabers_examples indexes the installed standalone examples and their annotated articles. Each collection has a contents list for navigating scripts: vignette("examples-manual", package = "vennabers"), vignette("examples-workflows", package = "vennabers"), vignette("examples-postprocessing", package = "vennabers") and vignette("examples-tuning", package = "vennabers"). Use system.file("examples", package = "vennabers") to locate the scripts for inspection or execution with source().

Assessment and interpretation

Score final classification probabilities with brier_score and log_loss, and assess them on held-out pairs with corp_plot, corp_decomposition_plot or calibration_plot; assess regression predictions with regression_plot. plot() draws the fit diagnosis of a fitted calibrator from its stored state, without new truth values. vignette("diagnostics", package = "vennabers") compares both kinds of graphics. Keep assessment data separate from model training, selection and calibration. Manual Cross calibration requires predictions from the corresponding retained fold models.

Classification bounds are Venn-Abers probability bounds, not confidence intervals. Regression bounds concern regularized conditional expectations, not individual outcome coverage; Cross aggregate bounds are descriptive.

Examples

calibrator <- venn_abers(
  truth = c(FALSE, TRUE, FALSE, TRUE),
  probability = c(0.1, 0.4, 0.35, 0.8)
)
prediction <- predict(calibrator, c(0.2, 0.5, 0.9))
prediction
#>       lower upper probability
#> 1 0.0000000   0.5   0.3333333
#> 2 0.5000000   1.0   0.6666667
#> 3 0.6666667   1.0   0.7500000
brier_score(c(FALSE, TRUE, TRUE), prediction$probability)
#> [1] 0.09490741