Calculate the average precision

The average_precision function returns the average precision of every precision-recall curve of an S3 object generated by evalmod(). Average precision summarizes a precision-recall curve as the precision at each cutoff weighted by the recall it gains over the cutoff before it.

Usage

average_precision(curves)

# S3 method for class 'aucs'
average_precision(curves)

Arguments

curves

An S3 object generated by evalmod().

Value

The average_precision function returns a data frame with the columns modnames, dsids and aps. Unlike auc() it has one row per model and dataset rather than one per curve, because average precision is defined on the precision-recall curve only.

How this differs from the area under the curve

Average precision is \(AP = \sum_i (r_i - r_{i-1}) p_i\), summed over the cutoffs from the second one on. Joining the raw precision-recall points with horizontal steps is not the same as measuring the area under the curve through them, and it is not the same number: the step estimator reads high wherever the two disagree, because linear or stepwise movement between precision-recall points overstates what a classifier can actually achieve there.

auc() reports the area under the curve evalmod() has interpolated the correct way, which is the reason this package exists, and is the number to prefer. Average precision is reported because several other packages report it under this name, and because the size of the gap between the two is itself worth seeing.

The sum starts at the second cutoff, so the precision of the empty prediction set - 0/0, which every tool defines differently - never enters it.

See also

auc() for the areas under the ROC and precision-recall curves. prbe() for the precision-recall break-even point. evalmod() for generating S3 objects with performance evaluation measures.

Examples

##################################################
### Single model & single test dataset
###
samps <- create_sim_samples(1, 50, 50, "good_er")
sscurves <- evalmod(scores = samps[["scores"]], labels = samps[["labels"]])
average_precision(sscurves)

## The area under the properly interpolated curve, for comparison
auc(sscurves)
#>   modnames dsids curvetypes      aucs
#> 1       m1     1        ROC 0.8648000
#> 2       m1     1        PRC 0.8978258

##################################################
### Multiple models & multiple test datasets
###
samps <- create_sim_samples(4, 50, 50, "all")
mdat <- mmdata(samps[["scores"]], samps[["labels"]],
  modnames = samps[["modnames"]], dsids = samps[["dsids"]]
)
mmcurves <- evalmod(mdat)
average_precision(mmcurves)