The average_precision function returns the average precision of
every precision-recall curve of an S3 object generated by
evalmod(). Average precision summarizes a precision-recall curve as
the precision at each cutoff weighted by the recall it gains over the
cutoff before it.
Arguments
- curves
An
S3object generated byevalmod().
Value
The average_precision function returns a data frame with the
columns modnames, dsids, aps and baselines. Unlike auc() it
has one row per model and dataset rather than one per curve, because
average precision is defined on the precision-recall curve only.
baselines is the proportion of positives, which is what average
precision is worth by chance - see
Reading an area against its baseline in auc(). It moves with the
class balance, so an average precision quoted without it does not say
how good the classifier is.
How this differs from the area under the curve
Average precision is \(AP = \sum_i (r_i - r_{i-1}) p_i\), summed over the cutoffs from the second one on. Joining the raw precision-recall points with horizontal steps is not the same as measuring the area under the curve through them, and it is not the same number: the step estimator reads high wherever the two disagree, because linear or stepwise movement between precision-recall points overstates what a classifier can actually achieve there.
auc() reports the area under the curve evalmod() has interpolated
the correct way, which is the reason this package exists, and is the
number to prefer. Average precision is reported because several other
packages report it under this name, and because the size of the gap
between the two is itself worth seeing.
The sum starts at the second cutoff, so the precision of the empty
prediction set - 0/0, which every tool defines differently - never
enters it.
Examples
##################################################
### Single model & single test dataset
###
samps <- create_sim_samples(1, 50, 50, "good_er")
sscurves <- evalmod(scores = samps[["scores"]], labels = samps[["labels"]])
average_precision(sscurves)
#> modnames dsids aps baselines
#> 1 m1 1 0.8406527 0.5
## The area under the properly interpolated curve, for comparison
auc(sscurves)
#> modnames dsids curvetypes aucs baselines
#> 1 m1 1 ROC 0.8216000 0.5
#> 2 m1 1 PRC 0.8387735 0.5
##################################################
### Multiple models & multiple test datasets
###
samps <- create_sim_samples(4, 50, 50, "all")
mdat <- mmdata(samps[["scores"]], samps[["labels"]],
modnames = samps[["modnames"]], dsids = samps[["dsids"]]
)
mmcurves <- evalmod(mdat)
average_precision(mmcurves)
#> modnames dsids aps baselines
#> 1 random 1 0.5190845 0.5
#> 2 poor_er 1 0.7118629 0.5
#> 3 good_er 1 0.8088922 0.5
#> 4 excel 1 0.9829871 0.5
#> 5 perf 1 1.0000000 0.5
#> 6 random 2 0.5538369 0.5
#> 7 poor_er 2 0.7903907 0.5
#> 8 good_er 2 0.8726102 0.5
#> 9 excel 2 0.9756416 0.5
#> 10 perf 2 1.0000000 0.5
#> 11 random 3 0.6000141 0.5
#> 12 poor_er 3 0.7315984 0.5
#> 13 good_er 3 0.8394689 0.5
#> 14 excel 3 0.9728387 0.5
#> 15 perf 3 1.0000000 0.5
#> 16 random 4 0.5339377 0.5
#> 17 poor_er 4 0.7260105 0.5
#> 18 good_er 4 0.7961333 0.5
#> 19 excel 4 0.9929833 0.5
#> 20 perf 4 1.0000000 0.5