Inspect Calibrated Workflow Predictions

Return native predictions together with calibration bounds, fold identities and source-model diagnostics from a fitted trainable or frozen workflow. Prediction uses the retained models and calibrators without learning again.

Usage

venn_abers_details(object, new_data, ..., loss = NULL)

Arguments

object: A fitted workflow returned by calibrated_workflow and generics::fit(), or by calibrate_workflow.
new_data: A data frame with required raw predictors. The same blueprint, schema and inner recipe/engine checks apply as for predict().
...: Must be empty. Unknown, irrelevant and partially matched names are errors.
loss: For classification, omit to use the stored default, or supply "log" or "brier" to override it without refitting or changing stored state. Explicit NULL is invalid. Regression rejects explicit loss.

Value

venn_abers_details() returns a list with prediction (the same native probability table) and details: integer row, character fold ("Inductive1" for Inductive; the supplied fold IDs for Cross), character event_class, and double lower, upper, probability. Cross details contain one row per new observation and fold; their probability is the fold point, while prediction contains the final aggregate. Trainable workflow details also contain character prediction_model and model_source. The source is the fold ID in matched mode, or "Full1" for every fold in full mode. These are model metadata, separate from original class identities and the fold identifying the held-out calibrator; venn_abers_decode() preserves them without decoding them as classes. Trainable OvO/OvR details include model_training. OvO/OvR details additionally contain character method and either class_1/class_2 (OvO) or class (OvR). Each row identifies one new observation, fold and binary subproblem. event_class is the later class in an OvO pair or the considered OvR class. These binary point probabilities must first be merged across folds for Cross (Log/Brier), then coupled (OvO) or normalized (OvR) to reconstruct prediction. Regression returns the native .pred table as prediction, and details with integer row, character fold, and double prediction, lower, upper. Cross additionally returns aggregate with row, lower, upper: descriptive separate equal-weight means of fold bounds. Trainable regression aggregates also contain character prediction_model and a model_source list column: each row lists all contributing source model IDs (only "Full1" in full mode). Zero/one rows retain these schemas. Frozen regression outputs retain their existing schema.

Postprocessing diagnostics

For learned postprocessing, the additional postprocessing element records local data memberships, fitted mappings and fit counts. Pairwise OvO uses unique pair_N/VA_fold keys; ensemble prediction details retain inner-fold names and sizes in list-columns. See workflow_postprocessing for the complete record layout and method-specific diagnostics.

Interpreting bounds

Classification bounds are Venn-Abers probability intervals, not confidence intervals. Multiclass details describe binary subproblems, not multiclass intervals. Regression bounds concern regularized conditional expectations, not individual outcome coverage; Cross aggregate bounds are descriptive.

Examples

packages <- c("parsnip", "workflows", "recipes", "rsample", "hardhat",
              "generics", "rlang", "tibble", "tidyselect", "withr")
if (all(vapply(packages, requireNamespace, logical(1), quietly = TRUE))) {
  development <- data.frame(
    x = sin(1:80),
    truth = factor(rep(c("yes", "no", "no", "yes"), 20),
                   levels = c("yes", "no"))
  )
  base <- workflows::workflow() |>
    workflows::add_formula(truth ~ x) |>
    workflows::add_model(parsnip::logistic_reg(engine = "glm"))
  specification <- calibrated_workflow(base, outcome = "truth",
                                       method = "cross", v = 3, seed = 42)
  fitted <- generics::fit(specification, development)
  new_data <- data.frame(x = c(-0.5, 0.5))

  # The final probabilities, then one record per new row and Cross fold.
  result <- venn_abers_details(fitted, new_data)
  print(result$prediction)
  result$details
}
#> # A tibble: 2 × 2
#>   .pred_yes .pred_no
#>       <dbl>    <dbl>
#> 1     0.493    0.507
#> 2     0.521    0.479
#> # A tibble: 6 × 8
#>     row fold  event_class lower upper probability prediction_model model_source
#>   <int> <chr> <chr>       <dbl> <dbl>       <dbl> <chr>            <chr>       
#> 1     1 Fold1 yes         0.429 0.476       0.455 matched          Fold1       
#> 2     2 Fold1 yes         0.429 0.476       0.455 matched          Fold1       
#> 3     1 Fold2 yes         0.526 0.6         0.559 matched          Fold2       
#> 4     2 Fold2 yes         0.5   0.579       0.537 matched          Fold2       
#> 5     1 Fold3 yes         0.429 0.5         0.467 matched          Fold3       
#> 6     2 Fold3 yes         0.538 0.615       0.571 matched          Fold3