The auc function takes an S3 object generated by
evalmod() and retrieves a data frame with the Area Under
the Curve (AUC) scores of ROC and Precision-Recall curves.
Arguments
- curves
An
S3object generated byevalmod(). Theaucfunction accepts the following S3 objects.S3object# of models # of test datasets sscurves single single mscurves multiple single smcurves single multiple mmcurves multiple multiple See the Value section of
evalmod()for more details.- macro
A Boolean value to specify whether the macro-average of the per-class AUCs is added. It is effective only for a multiclass evaluation - see the
multiclassargument ofmmdata()- and the added rows carrymacro-averageas their model name. Classes that could not be evaluated are left out of the average.- macro_weight
How the per-class AUCs are weighted in that average.
"uniform", the default, gives every class the same weight, so a rare class counts as much as a common one."prevalence"weights each class by the number of observations it has, so the average follows the class distribution of the data. The weighted rows are namedmacro-average-weightedto keep the two apart. The two agree on a balanced dataset.For a ROC evaluation the two are the metrics other packages call
roc_aunuandroc_aunprespectively.
Value
The auc function returns a data frame with one row per curve
per model per test dataset, and the following columns.
modnames | Model name |
dsids | Test dataset ID |
curvetypes | ROC or PRC |
aucs | The area under that curve |
baselines | What that area would be by chance, see below |
Reading an area against its baseline
A ROC curve's chance level is 0.5 whatever the data, so a ROC AUC can
be read on its own. A precision-recall curve's chance level is the
proportion of positives, so a PRC AUC cannot: the same number means
different things on different data, and the difference is not small.
The baselines column carries that value, 0.5 on every ROC row and
the prevalence on every PRC row, so that the area and what it is
worth arrive together. On the same generator at three class balances:
| Positives | ROC AUC | PRC AUC | PRC baseline |
| 50% | 0.807 | 0.801 | 0.50 |
| 10% | 0.834 | 0.367 | 0.10 |
| 2% | 0.850 | 0.129 | 0.02 |
The classifier is about as good in all three rows and the ROC AUC says
so. The PRC AUC falls to 0.129, which reads as failure and is in fact
six times chance. Quote the two numbers together.
The baseline is looked up per model and per test dataset, because a
fold need not hold the classes in the proportions the whole dataset
does. On a macro-average row it is averaged over the classes exactly
as the AUCs are, with the same weights, so the row is still read
against the chance level of the mixture that produced it.
The baseline is an asymptote
The prevalence is what a precision-recall area is worth by chance in the limit. An area measured on a finite sample scatters around its chance level rather than sitting on it, and the fewer the positives the wider it scatters and the further above it the average sits. Dividing the two numbers is therefore not a test.
With twenty positives at two percent prevalence, a classifier with no
signal at all averages 1.15 times its baseline over the whole curve
and 1.91 times over the first tenth of recall, where it clears twice
the baseline 22% of the time. Two hundred positives brings the first
figure to 1.01. What drives it is the number of positives, not the
balance, and average_precision() shows the same thing, so it is not
an artifact of the interpolation.
The example above stands: 0.129 against a baseline of 0.02 is a
real result. But when the margin is small and the positives are few,
auc_boot() puts an interval around the area from a single test set,
and auc_ci() does it from several. Both now report the baseline
beside the interval, which is the comparison worth making.
See also
evalmod() for generating S3 objects with
performance evaluation metrics. pauc() for retrieving
a dataset of pAUCs. average_precision() for the step estimator of
the area under the precision-recall curve. auc_boot() and auc_ci()
for an interval around an area, which is what the gap between an area
and its baseline has to be read against.
Examples
##################################################
### Single model & single test dataset
###
## Load a dataset with 10 positives and 10 negatives
data(P10N10)
## Generate an sscurve object that contains ROC and Precision-Recall curves
sscurves <- evalmod(scores = P10N10$scores, labels = P10N10$labels)
## Shows AUCs
auc(sscurves)
#> modnames dsids curvetypes aucs baselines
#> 1 m1 1 ROC 0.7200000 0.5
#> 2 m1 1 PRC 0.7397716 0.5
##################################################
### Multiple models & single test dataset
###
## Create sample datasets with 100 positives and 100 negatives
samps <- create_sim_samples(1, 100, 100, "all")
mdat <- mmdata(samps[["scores"]], samps[["labels"]],
modnames = samps[["modnames"]]
)
## Generate an mscurve object that contains ROC and Precision-Recall curves
mscurves <- evalmod(mdat)
## Shows AUCs
auc(mscurves)
#> modnames dsids curvetypes aucs baselines
#> 1 random 1 ROC 0.4971000 0.5
#> 2 random 1 PRC 0.4992116 0.5
#> 3 poor_er 1 ROC 0.8328000 0.5
#> 4 poor_er 1 PRC 0.7860641 0.5
#> 5 good_er 1 ROC 0.8180000 0.5
#> 6 good_er 1 PRC 0.8574152 0.5
#> 7 excel 1 ROC 0.9780000 0.5
#> 8 excel 1 PRC 0.9782574 0.5
#> 9 perf 1 ROC 1.0000000 0.5
#> 10 perf 1 PRC 1.0000000 0.5
##################################################
### Single model & multiple test datasets
###
## Create sample datasets with 100 positives and 100 negatives
samps <- create_sim_samples(4, 100, 100, "good_er")
mdat <- mmdata(samps[["scores"]], samps[["labels"]],
modnames = samps[["modnames"]],
dsids = samps[["dsids"]]
)
## Generate an smcurve object that contains ROC and Precision-Recall curves
smcurves <- evalmod(mdat, raw_curves = TRUE)
## Get AUCs
sm_aucs <- auc(smcurves)
## Shows AUCs
sm_aucs
#> modnames dsids curvetypes aucs baselines
#> 1 good_er 1 ROC 0.7865000 0.5
#> 2 good_er 1 PRC 0.8404735 0.5
#> 3 good_er 2 ROC 0.8313000 0.5
#> 4 good_er 2 PRC 0.8628264 0.5
#> 5 good_er 3 ROC 0.8244000 0.5
#> 6 good_er 3 PRC 0.8578336 0.5
#> 7 good_er 4 ROC 0.8204000 0.5
#> 8 good_er 4 PRC 0.8519919 0.5
## Get AUCs of Precision-Recall
sm_aucs_prc <- subset(sm_aucs, curvetypes == "PRC")
## Shows AUCs
sm_aucs_prc
#> modnames dsids curvetypes aucs baselines
#> 2 good_er 1 PRC 0.8404735 0.5
#> 4 good_er 2 PRC 0.8628264 0.5
#> 6 good_er 3 PRC 0.8578336 0.5
#> 8 good_er 4 PRC 0.8519919 0.5
##################################################
### Multiple models & multiple test datasets
###
## Create sample datasets with 100 positives and 100 negatives
samps <- create_sim_samples(4, 100, 100, "all")
mdat <- mmdata(samps[["scores"]], samps[["labels"]],
modnames = samps[["modnames"]],
dsids = samps[["dsids"]]
)
## Generate an mscurve object that contains ROC and Precision-Recall curves
mmcurves <- evalmod(mdat, raw_curves = TRUE)
## Get AUCs
mm_aucs <- auc(mmcurves)
## Shows AUCs
mm_aucs
#> modnames dsids curvetypes aucs baselines
#> 1 random 1 ROC 0.4509000 0.5
#> 2 random 1 PRC 0.4468330 0.5
#> 3 poor_er 1 ROC 0.8299000 0.5
#> 4 poor_er 1 PRC 0.7817533 0.5
#> 5 good_er 1 ROC 0.8285000 0.5
#> 6 good_er 1 PRC 0.8577692 0.5
#> 7 excel 1 ROC 0.9820000 0.5
#> 8 excel 1 PRC 0.9842602 0.5
#> 9 perf 1 ROC 1.0000000 0.5
#> 10 perf 1 PRC 1.0000000 0.5
#> 11 random 2 ROC 0.4974000 0.5
#> 12 random 2 PRC 0.5102109 0.5
#> 13 poor_er 2 ROC 0.7718000 0.5
#> 14 poor_er 2 PRC 0.7117766 0.5
#> 15 good_er 2 ROC 0.7925000 0.5
#> 16 good_er 2 PRC 0.8071713 0.5
#> 17 excel 2 ROC 0.9778000 0.5
#> 18 excel 2 PRC 0.9778305 0.5
#> 19 perf 2 ROC 1.0000000 0.5
#> 20 perf 2 PRC 1.0000000 0.5
#> 21 random 3 ROC 0.4797000 0.5
#> 22 random 3 PRC 0.5184681 0.5
#> 23 poor_er 3 ROC 0.8219000 0.5
#> 24 poor_er 3 PRC 0.7939097 0.5
#> 25 good_er 3 ROC 0.7832000 0.5
#> 26 good_er 3 PRC 0.8267456 0.5
#> 27 excel 3 ROC 0.9797000 0.5
#> 28 excel 3 PRC 0.9824702 0.5
#> 29 perf 3 ROC 1.0000000 0.5
#> 30 perf 3 PRC 1.0000000 0.5
#> 31 random 4 ROC 0.4924000 0.5
#> 32 random 4 PRC 0.4723165 0.5
#> 33 poor_er 4 ROC 0.7803000 0.5
#> 34 poor_er 4 PRC 0.7185701 0.5
#> 35 good_er 4 ROC 0.8343000 0.5
#> 36 good_er 4 PRC 0.8609178 0.5
#> 37 excel 4 ROC 0.9891000 0.5
#> 38 excel 4 PRC 0.9890191 0.5
#> 39 perf 4 ROC 1.0000000 0.5
#> 40 perf 4 PRC 1.0000000 0.5
## Get AUCs of Precision-Recall
mm_aucs_prc <- subset(mm_aucs, curvetypes == "PRC")
## Shows AUCs
mm_aucs_prc
#> modnames dsids curvetypes aucs baselines
#> 2 random 1 PRC 0.4468330 0.5
#> 4 poor_er 1 PRC 0.7817533 0.5
#> 6 good_er 1 PRC 0.8577692 0.5
#> 8 excel 1 PRC 0.9842602 0.5
#> 10 perf 1 PRC 1.0000000 0.5
#> 12 random 2 PRC 0.5102109 0.5
#> 14 poor_er 2 PRC 0.7117766 0.5
#> 16 good_er 2 PRC 0.8071713 0.5
#> 18 excel 2 PRC 0.9778305 0.5
#> 20 perf 2 PRC 1.0000000 0.5
#> 22 random 3 PRC 0.5184681 0.5
#> 24 poor_er 3 PRC 0.7939097 0.5
#> 26 good_er 3 PRC 0.8267456 0.5
#> 28 excel 3 PRC 0.9824702 0.5
#> 30 perf 3 PRC 1.0000000 0.5
#> 32 random 4 PRC 0.4723165 0.5
#> 34 poor_er 4 PRC 0.7185701 0.5
#> 36 good_er 4 PRC 0.8609178 0.5
#> 38 excel 4 PRC 0.9890191 0.5
#> 40 perf 4 PRC 1.0000000 0.5
##################################################
### Multiclass evaluation
###
## Load a 3-class dataset with one score column per class
data(C3N150)
## One-vs-rest curves
mccurves <- evalmod(scores = C3N150$scores, labels = C3N150$labels)
## Per-class AUCs, plus their macro-average
auc(mccurves)
#> modnames dsids curvetypes aucs baselines
#> 1 c1 1 ROC 0.9732000 0.5000000
#> 2 c1 1 PRC 0.9558435 0.3333333
#> 3 c2 1 ROC 0.7758000 0.5000000
#> 4 c2 1 PRC 0.6550357 0.3333333
#> 5 c3 1 ROC 0.5336000 0.5000000
#> 6 c3 1 PRC 0.4162555 0.3333333
#> 7 macro-average 1 ROC 0.7608667 0.5000000
#> 8 macro-average 1 PRC 0.6757116 0.3333333
## Per-class AUCs only
auc(mccurves, macro = FALSE)
#> modnames dsids curvetypes aucs baselines
#> 1 c1 1 ROC 0.9732000 0.5000000
#> 2 c1 1 PRC 0.9558435 0.3333333
#> 3 c2 1 ROC 0.7758000 0.5000000
#> 4 c2 1 PRC 0.6550357 0.3333333
#> 5 c3 1 ROC 0.5336000 0.5000000
#> 6 c3 1 PRC 0.4162555 0.3333333
## Weighted by the class distribution instead
auc(mccurves, macro_weight = "prevalence")
#> modnames dsids curvetypes aucs baselines
#> 1 c1 1 ROC 0.9732000 0.5000000
#> 2 c1 1 PRC 0.9558435 0.3333333
#> 3 c2 1 ROC 0.7758000 0.5000000
#> 4 c2 1 PRC 0.6550357 0.3333333
#> 5 c3 1 ROC 0.5336000 0.5000000
#> 6 c3 1 PRC 0.4162555 0.3333333
#> 7 macro-average-weighted 1 ROC 0.7608667 0.5000000
#> 8 macro-average-weighted 1 PRC 0.6757116 0.3333333