Score Classification Probabilities — brier

Computes the Brier Score or Log Loss from final binary or multiclass probabilities, regardless of the loss chosen during calibration prediction.

Usage

brier_score(truth, probability)

log_loss(truth, probability)

Arguments

truth: A non-empty logical or numeric vector containing only FALSE/0 and TRUE/1. TRUE/1 is the event class. Alternatively a non-empty factor with at least two non-empty levels and no missing values. Assessment subsets may omit observations of some levels; retain the full fitted class schema.
probability: A numeric vector of regularized event probabilities, the same length as truth, with finite values in [0, 1]. For factor truth, a numeric class matrix following the class-name, value, and row-sum contract of venn_abers.

Value

One unnamed double. brier_score() returns the binary mean squared error. log_loss() returns the mean negative natural log-likelihood; an incorrect probability of zero or one produces Inf. For multiclass, Brier is the mean of the row sums of squared errors against one-hot truth, without division by two or class count. A two-column Brier score is twice the binary vector score for the same probabilities. Multiclass log loss is the mean negative natural logarithm of the observed class probability; zero observed probability gives Inf. Neither score clips probabilities.

Examples

truth <- c(FALSE, TRUE, FALSE, TRUE)
uncalibrated_probability <- c(0.1, 0.4, 0.35, 0.8)
calibrator <- venn_abers(truth, uncalibrated_probability)
prediction <- predict(calibrator, uncalibrated_probability)

brier_score(truth, prediction$probability)
#> [1] 0.08680556
log_loss(truth, prediction$probability)
#> [1] 0.3465736