Train a Calibrated Classification or Regression Workflow

Wraps an unfitted tidymodels workflow for classification or regression so that generics::fit() trains preprocessing, models and a Venn-Abers calibrator on separate local splits of the same data. It needs the workflow and the outcome column; the fitted workflow is then used with predict() on raw predictors. Choose it when the package should fit the model and keep the calibration rows out of training; use calibrate_workflow for a workflow that is already fitted, or venn_abers and venn_abers_regression for predictions computed elsewhere.

Usage

calibrated_workflow(workflow, outcome, ..., predictors = NULL,
  method = "inductive", classification = "binary", event_level = "first",
  calibration_prop = 0.2, training_size = NULL, calibration_size = NULL,
  v = 5L, strata = NULL, splits = NULL, seed = NULL,
  precision = NULL, loss = "log", m = NULL, epsilon = NULL, schema = NULL,
  prediction_model = "matched", model_training = "shared", tuning = NULL,
  postprocessing = NULL)
# S3 method for class '`_vennabers_fit`'
predict(object, new_data, type = NULL,
  opts = list(), ..., loss = NULL)

Arguments

workflow: An unfitted workflows object with a classification or regression model and recipe, formula, or variable selection. Prepped or partly trained recipes, unresolved tune placeholders without tuning, and postprocessors without an explicit matching postprocessing request are rejected.
outcome: One original outcome column name, or schema$encoded_outcome when a schema is supplied. Fit requires a nonmissing factor with exactly two levels for binary or at least two for OvO/OvR. All stored levels must be represented in the development data. Regression requires finite numeric truth; numeric 0/1 outcomes remain regression.
...: Must be empty. Unknown, irrelevant and partially matched names are errors.
predictors: Optional character vector of raw predictor names, excluding outcome. NULL selects all other supplied columns. Their stored order is canonical; unused selected columns still have to be supplied for prediction.
method: Exactly "inductive" (default) or "cross".
classification: Exactly "binary" (default), "ovo", or "ovr". OvO/OvR explicitly use the factor/matrix core, including with two classes. Mode comes from the base model. Regression rejects explicit classification, event_level, precision and loss options, including their defaults; implicit classification defaults do not apply.
event_level: Exactly "first" or "second", identifying the original factor level whose probability is calibrated. Only for binary; explicitly supplying this option for OvO/OvR is an error.
calibration_prop: A number strictly between zero and one. rsample::initial_split() uses training proportion 1-calibration_prop. Explicitly supplying this option requires Inductive.
training_size, calibration_size: Independent Inductive training and calibration sizes, like sklearn's train_size/test_size. Both NULL (default) keep calibration_prop and its draw unchanged. A value strictly between zero and one is a proportion of the local rows; a whole number of at least one is a row count. Proportions round training down and calibration up; row counts are exact. A missing size is the complement. Rows left over when both are supplied form the unassigned set U: the matched source model, its B/P postprocessing and the calibrator never use them, while prediction_model = "full" trains its additional workflow on all local rows, including calibration rows and U. Classification draws are stratified by outcome with exact totals (largest-remainder class allocation, ties in level order) unless strata = FALSE; regression is unstratified. Sizes are checked against the local rows and never shrunk: more rows than available, or an empty training or calibration set, is an error naming both sizes. They cannot be tuned and cannot be combined with method = "cross", explicit calibration_prop or splits; the default calibration_prop is then ignored. With model_training = "pairwise" the sizes resolve on each class pair's local rows, with U per pair; after calibration selection they resolve on the final-fit F. Calibration tuning and joint model tuning fit Venn-Abers candidates that resolve the same sizes on their resample rows, so they accept only proportions; two-stage model tuning also accepts row counts. See inductive-sizes.R in vennabers_examples.
v: Cross fold count, a whole number of at least two, default five. Automatic rsample::vfold_cv() uses one repetition and never reduces this count; it must not exceed the local row count. Explicitly supplying v requires Cross.
strata: NULL stratifies classification by outcome with pool=0; regression is unstratified. FALSE disables stratification. Other values are rejected. No retry or automatic split adjustment.
splits: NULL generates a local split. Otherwise a mixed or ordered holdout rsplit from rsample::initial_split() or rsample::make_splits(), for Inductive; a simple rset from rsample::vfold_cv() or rsample::manual_rset() for Cross; or a serializable function receiving only selected local raw predictors followed by outcome and returning such a plan. Explicit plans cannot be combined with explicit calibration_prop, v or strata, even their defaults. Use local_calibration_splits for prepared assignments. Pairwise accepts a function of the current local pair data, or for direct fits a list named by exactly pair_1, pair_2, etc. A single global multiclass plan is not a pair plan. Automatic holdouts/partitions are generated after filtering. Explicit Cross plans may have different fold counts per pair.
seed: NULL uses the caller's RNG stream. A finite whole number in R's integer seed range sets a local seed for splitting and base fitting, restoring the caller's state and RNG kind afterwards, including on error. The scope includes the plan callback and model learning. Seed the outer plan separately. Reproducibility also depends on RNG kind, engine, package versions, thread and parallel configuration.
precision: NULL or a finite numeric whole number, including negative numbers. Only calibration probabilities are rounded, following venn_abers().
loss: Constructor: stored prediction default, exactly "log" or "brier". Prediction/details: omit to use that default; an explicit choice overrides it without refitting or changing stored state. Explicit NULL is invalid. Regression rejects explicit loss at construction and prediction/details.
m, epsilon: Regression moderation: both NULL means effective m=1. Only one non-NULL option may be supplied and only that option reaches the core. m is a positive numeric whole number; epsilon is strictly between zero and one. Before any inner training, each fold of calibration size k must satisfy 1 <= m <= floor((k-1)/2), with m = floor(epsilon*(k+1)/2) for an epsilon request. Invalid folds fail the entire fit with fold/size context; there is no clipping or fold-count adjustment. Classification rejects explicit m/epsilon, including NULL.
schema: NULL retains the direct naming contract. A schema from venn_abers_schema() fixes the encoded input names, mode, class order and binary event. Omitted classification/event options inherit the schema; contradictory explicit options fail before learning. Build the base formula/recipe and resamples on venn_abers_encode() data. All schema predictors must be selected.
prediction_model: Exactly "matched" (default) or "full". Available for binary, shared OvO/OvR, pairwise OvO classification and regression. This is a fixed fit option; predict cannot change it. Matched uses each calibration source model for new observations. Full applies the unchanged held-out calibrators to predictions from one additional workflow trained on the entire permitted local fit domain, including calibration rows. For pairwise training there is one such full workflow per local class pair. With postprocessing, full means an additional complete B/P holdout or ensemble on that domain, with its own inner splits; each engine still learns only on B. Matching-model calibration guarantees do not automatically transfer to full models; see workflow_training.
model_training: Exactly "shared" (default) or "pairwise". Explicit values are rejected for binary and regression workflows. A shared model produces one complete class matrix; OvO splits calibration into pairs without training pair models. Pairwise requires classification="ovo", and a workflow that can train on a two-level factor. Each pair has its own models, recipes and holdout or Cross partition. The later global level is its event; no extra shared/OvR model is fitted.
tuning: NULL keeps fixed settings. Otherwise, a named list with strategy = "joint" or "two_stage", and model and/or calibration Grid or Bayes searches matching the tune() placeholders. See venn_abers_tuning for the configuration, selection splits, pairwise search rules and diagnostics; start with vignette("tuning", package = "vennabers") for examples.
postprocessing: NULL leaves the unprocessed training path unchanged; it does not remove a supplied postprocessor. Otherwise, a named list selecting method ("sigmoid", "tailor_logistic", "tailor_multinomial", "tailor_linear" or "isotonic_numeric") and strategy ("holdout" or "ensemble"). Holdouts need prop or local splits; ensembles need v or local splits. Sigmoid also requires an explicit sigmoid_input. See workflow_postprocessing for compatible methods, data roles and the complete configuration.
object: A fitted calibrated workflow. The predict method is normally dispatched by predict() on that workflow.
new_data: A data frame with required raw predictors. Normal workflow blueprint and hardhat::forge() checks apply equally to prediction and details. Novel factor levels reach the inner workflow, whose recipe/engine handles them or errors.
type: Classification: "prob" or "class", default class. Regression: "numeric", also the default, returns double .pred in outcome units. Neither "conf_int" nor "pred_int" is registered.
opts: The standard parsnip prediction options list; normally empty.

Value

The constructor returns an unfitted ordinary workflow. After generics::fit(), native probability predictions are tibbles of double .pred_<original class> columns in original level order. Class predictions contain .pred_class, a factor retaining all levels. Binary favors its event at exactly 0.5; OvO/OvR select the first maximum in stored level order from the final probabilities, even with two classes. Empty predictions keep the schema without calling the inner engine. venn_abers_one_hot represents these class decisions as a one-hot matrix.

Regression predictions contain a double .pred column in outcome units. Use venn_abers_details for bounds, fold identities and source models, and venn_abers_tuning for selection diagnostics. The fitted engine, workflows::extract_fit_engine(fitted), is a vennabers_fit: printing it shows source mappings and base-fit counts, and plot.vennabers_fit draws its stored calibration map.

Learning path

Start with the binary example below, then follow vignette("workflows", package = "vennabers") through learned preprocessing, Cross calibration, multiclass, regression, frozen workflows and postprocessing. vignette("tuning", package = "vennabers") continues with parameter selection, outer resampling and final assessment.

For individual rules, use these reference topics:

Training and evaluation: workflow_training: dependencies, split validation, stored state, pairwise models, full-model transfer and resampling.
Prediction diagnostics: venn_abers_details: returned columns, bounds and fold/source identities.
Tuning: venn_abers_tuning: search configuration, selection data, local searches and diagnostics.
Postprocessing: workflow_postprocessing: supported methods, B/P/C data roles, holdouts, ensembles and full compositions.

For a model that is already fitted, use calibrate_workflow.

Data roles and limitations

The split isolates new base learning from calibration observations. An unfitted workflow does not prove that earlier tuning, feature choices, or external transformations excluded these observations. Keep assessment observations separate from all training, selection and calibration. Calibration is not an independent evaluation. Cross model parameters must be fixed using independent data or selected through the configured local tuning searches inside the allowed training portions. The statistical setting assumes IID observations and a predictive procedure fixed independently of its calibration data. The bounds are Venn-Abers probability intervals, not confidence intervals. Regression bounds concern regularized conditional expectations, not individual outcome coverage; Cross aggregate bounds are descriptive. Improved scores on every data set are not promised. The engine cannot validate an outer partition it never receives. Validate outer analysis/assessment disjointness before calling fit_resamples; assessment sets from different outer resamples may overlap.

Examples

packages <- c("parsnip", "workflows", "recipes", "rsample", "hardhat",
              "generics", "rlang", "tibble", "tidyselect", "withr")
if (all(vapply(packages, requireNamespace, logical(1), quietly = TRUE))) {
  # Development rows are available for training and calibration.
  # Reserve separate assessment data before choosing this procedure.
  development <- data.frame(
    x = sin(1:80),
    truth = factor(rep(c("yes", "no", "no", "yes"), 20),
                   levels = c("yes", "no"))
  )
  base <- workflows::workflow() |>
    workflows::add_formula(truth ~ x) |>
    workflows::add_model(parsnip::logistic_reg(engine = "glm"))

  specification <- calibrated_workflow(base, outcome = "truth", seed = 42)
  fitted <- generics::fit(specification, development)
  new_data <- data.frame(x = c(-0.5, 0.5))
  predict(fitted, new_data, type = "prob")
  venn_abers_details(fitted, new_data)$details
}
#> # A tibble: 2 × 8
#>     row fold   event_class lower upper probability prediction_model model_source
#>   <int> <chr>  <chr>       <dbl> <dbl>       <dbl> <chr>            <chr>       
#> 1     1 Induc… yes           0.5 0.571       0.533 matched          Inductive1  
#> 2     2 Induc… yes           0.5 0.571       0.533 matched          Inductive1