The pauc function takes an S3 object generated by
part() and evalmod() and retrieves a data frame
with the partial AUC scores of ROC and Precision-Recall curves.
Arguments
- curves
An
S3object generated bypart()andevalmod(). Thepaucfunction accepts the following S3 objects.S3object# of models # of test datasets sscurves single single mscurves multiple single smcurves single multiple mmcurves multiple multiple See the Value section of
evalmod()for more details.- corrected
A logical value to add the
cpaucscolumn, the McClish correction of the ROC partial areas. The default isFALSE.
Value
The pauc function returns a data frame with one row per curve
per model per test dataset, and the following columns. An object that
holds averaged curves only has no dsids column, because an averaged
curve has no single test dataset behind it.
modnames | Model name |
dsids | Test dataset ID |
curvetypes | ROC or PRC |
paucs | The area under that curve over the region |
baselines | What that area would be by chance, see below |
spaucs | That area over the area of the region |
sbaselines | What spaucs would be by chance |
cpaucs | The McClish correction, with corrected = TRUE |
Each area is followed by what it is worth by chance, because neither of
them can be read without it. cpaucs has no column of its own: the
correction puts chance at 0.5 by construction, which is the whole
point of it.
What a partial area is worth by chance
Restricting the region changes the chance level, and not by an amount
anyone guesses correctly. Over false positive rates [x1, x2] a coin
flip covers the area under the diagonal across that span,
(x2^2 - x1^2) / 2. A precision-recall curve is flat at the proportion
of positives instead, so chance covers prevalence * (x2 - x1).
spaucs divides by the area of the region, which divides those two by
the width: chance for a standardized ROC partial area is (x1 + x2) / 2
and not 0.5. Over false positive rates up to 0.2 a coin flip
covers 0.1 of the region, so a spaucs of 0.3 there is three times
chance and not a failing grade. For a precision-recall curve the
standardized chance level is the prevalence, unchanged by the region.
Both are NA when part() was given a ylim other than c(0, 1).
The region is then a box the chance curve may cross, touch or miss
entirely, and the area of chance inside it has no settled definition -
the case cpaucs declines for the same reason.
A baseline is what an area is worth by chance in the limit, and the
estimate from a finite sample scatters around it. Comparing an area to
its baseline is not a test, and the smaller the region and the fewer
the positives the less it is worth: over the first tenth of recall with
twenty positives, a classifier with no signal averages nearly twice its
baseline. auc_boot() puts an interval around the area, which is the
comparison that means something.
Two ways to standardize a partial area
A partial area is not comparable across regions on its own - a wider region holds more area - so it is reported standardized, and there are two conventions for doing it. Both are called "the standardized partial AUC", and a number carried between tools looks like a disagreement when it is a choice of convention.
spaucs divides the area by the area of the region, so it says what
fraction of what was available the curve covered. The McClish
correction instead rescales the span between chance and perfect onto
0.5 to 1, so that a partial area reads on the same scale as a full
one. Over false positive rates up to 0.2, a coin flip covers an area
of 0.02: that is 0.1 of the region, and 0.5 after the correction.
Neither is the right one. spaucs is the more direct reading of how
much of the corner the curve filled; the corrected one is the more
comparable to a full AUC. Say which you used.
cpaucs is what pROC::auc(..., partial.auc.correct = TRUE) returns,
to the last digit. The chance area is taken over the region actually
asked for, (x2^2 - x1^2) / 2, and not over a region assumed to start
at a false positive rate of 0 - the formula usually quoted is the
special case of that one, and gets a region such as
xlim = c(0.1, 0.3) wrong.
The correction is the identity over the whole curve, where chance is
0.5 and the region is 1, so a cpaucs from part(xlim = c(0, 1))
is the plain AUC. At the other end it is unstable: over a narrow region
of high false positive rates, chance and perfect are close together and
the rescaling divides by the small difference, so a curve a little
below the diagonal there can produce a large negative number. That is
the correction behaving as defined, not an error.
cpaucs is NA on every PRC row, and on every row when part() was
given a ylim other than c(0, 1). A precision-recall curve's chance
level is the proportion of positives rather than the diagonal, and
rescaling by it would make curves at different class balances look
comparable when the point of the precision-recall plot is that they are
not - see auc(). A restricted ylim has no counterpart in pROC and
no settled definition, so it reports nothing rather than invent one.
Examples
##################################################
### Single model & single test dataset
###
## Load a dataset with 10 positives and 10 negatives
data(P10N10)
## Generate an sscurve object that contains ROC and Precision-Recall curves
sscurves <- evalmod(scores = P10N10$scores, labels = P10N10$labels)
## Calculate partial AUCs
sscurves.part <- part(sscurves, xlim = c(0.25, 0.75))
## Shows pAUCs
pauc(sscurves.part)
#> modnames dsids curvetypes paucs baselines spaucs sbaselines
#> 1 m1 1 ROC 0.3771875 0.25 0.7543750 0.5
#> 2 m1 1 PRC 0.3616708 0.25 0.7233417 0.5
##################################################
### Multiple models & single test dataset
###
## Create sample datasets with 100 positives and 100 negatives
samps <- create_sim_samples(1, 100, 100, "all")
mdat <- mmdata(samps[["scores"]], samps[["labels"]],
modnames = samps[["modnames"]]
)
## Generate an mscurve object that contains ROC and Precision-Recall curves
mscurves <- evalmod(mdat)
## Calculate partial AUCs
mscurves.part <- part(mscurves, xlim = c(0, 0.75), ylim = c(0.25, 0.75))
## Shows pAUCs
pauc(mscurves.part)
#> modnames dsids curvetypes paucs baselines spaucs sbaselines
#> 1 random 1 ROC 0.1683000 NA 0.4488000 NA
#> 2 random 1 PRC 0.2409832 NA 0.6426219 NA
#> 3 poor_er 1 ROC 0.3086000 NA 0.8229333 NA
#> 4 poor_er 1 PRC 0.3736987 NA 0.9965299 NA
#> 5 good_er 1 ROC 0.3526000 NA 0.9402667 NA
#> 6 good_er 1 PRC 0.3750000 NA 1.0000000 NA
#> 7 excel 1 ROC 0.3737000 NA 0.9965333 NA
#> 8 excel 1 PRC 0.3750000 NA 1.0000000 NA
#> 9 perf 1 ROC 0.3750000 NA 1.0000000 NA
#> 10 perf 1 PRC 0.3750000 NA 1.0000000 NA
##################################################
### Single model & multiple test datasets
###
## Create sample datasets with 100 positives and 100 negatives
samps <- create_sim_samples(4, 100, 100, "good_er")
mdat <- mmdata(samps[["scores"]], samps[["labels"]],
modnames = samps[["modnames"]],
dsids = samps[["dsids"]]
)
## Generate an smcurve object that contains ROC and Precision-Recall curves
smcurves <- evalmod(mdat, raw_curves = TRUE)
## Calculate partial AUCs
smcurves.part <- part(smcurves, xlim = c(0.25, 0.75))
## Shows pAUCs
pauc(smcurves.part)
#> modnames dsids curvetypes paucs baselines spaucs sbaselines
#> 1 good_er 1 ROC 0.4212000 0.25 0.8424000 0.5
#> 2 good_er 1 PRC 0.4534013 0.25 0.9068026 0.5
#> 3 good_er 2 ROC 0.4120000 0.25 0.8240000 0.5
#> 4 good_er 2 PRC 0.4222832 0.25 0.8445664 0.5
#> 5 good_er 3 ROC 0.4056000 0.25 0.8112000 0.5
#> 6 good_er 3 PRC 0.4327445 0.25 0.8654890 0.5
#> 7 good_er 4 ROC 0.4218000 0.25 0.8436000 0.5
#> 8 good_er 4 PRC 0.4352211 0.25 0.8704422 0.5
##################################################
### Multiple models & multiple test datasets
###
## Create sample datasets with 100 positives and 100 negatives
samps <- create_sim_samples(4, 100, 100, "all")
mdat <- mmdata(samps[["scores"]], samps[["labels"]],
modnames = samps[["modnames"]],
dsids = samps[["dsids"]]
)
## Generate an mscurve object that contains ROC and Precision-Recall curves
mmcurves <- evalmod(mdat, raw_curves = TRUE)
## Calculate partial AUCs
mmcurves.part <- part(mmcurves, xlim = c(0, 0.25))
## Shows pAUCs
pauc(mmcurves.part)
#> modnames dsids curvetypes paucs baselines spaucs sbaselines
#> 1 random 1 ROC 0.0206000 0.03125 0.0824000 0.125
#> 2 random 1 PRC 0.1187358 0.12500 0.4749431 0.500
#> 3 poor_er 1 ROC 0.0958000 0.03125 0.3832000 0.125
#> 4 poor_er 1 PRC 0.2247158 0.12500 0.8988631 0.500
#> 5 good_er 1 ROC 0.1220000 0.03125 0.4880000 0.125
#> 6 good_er 1 PRC 0.2483295 0.12500 0.9933178 0.500
#> 7 excel 1 ROC 0.2362000 0.03125 0.9448000 0.125
#> 8 excel 1 PRC 0.2500000 0.12500 1.0000000 0.500
#> 9 perf 1 ROC 0.2500000 0.03125 1.0000000 0.125
#> 10 perf 1 PRC 0.2500000 0.12500 1.0000000 0.500
#> 11 random 2 ROC 0.0485000 0.03125 0.1940000 0.125
#> 12 random 2 PRC 0.1761329 0.12500 0.7045316 0.500
#> 13 poor_er 2 ROC 0.1201000 0.03125 0.4804000 0.125
#> 14 poor_er 2 PRC 0.2305095 0.12500 0.9220381 0.500
#> 15 good_er 2 ROC 0.1296000 0.03125 0.5184000 0.125
#> 16 good_er 2 PRC 0.2500000 0.12500 1.0000000 0.500
#> 17 excel 2 ROC 0.2332000 0.03125 0.9328000 0.125
#> 18 excel 2 PRC 0.2500000 0.12500 1.0000000 0.500
#> 19 perf 2 ROC 0.2500000 0.03125 1.0000000 0.125
#> 20 perf 2 PRC 0.2500000 0.12500 1.0000000 0.500
#> 21 random 3 ROC 0.0290000 0.03125 0.1160000 0.125
#> 22 random 3 PRC 0.1128633 0.12500 0.4514532 0.500
#> 23 poor_er 3 ROC 0.1116000 0.03125 0.4464000 0.125
#> 24 poor_er 3 PRC 0.2162563 0.12500 0.8650252 0.500
#> 25 good_er 3 ROC 0.1620000 0.03125 0.6480000 0.125
#> 26 good_er 3 PRC 0.2500000 0.12500 1.0000000 0.500
#> 27 excel 3 ROC 0.2346000 0.03125 0.9384000 0.125
#> 28 excel 3 PRC 0.2500000 0.12500 1.0000000 0.500
#> 29 perf 3 ROC 0.2500000 0.03125 1.0000000 0.125
#> 30 perf 3 PRC 0.2500000 0.12500 1.0000000 0.500
#> 31 random 4 ROC 0.0283000 0.03125 0.1132000 0.125
#> 32 random 4 PRC 0.1244835 0.12500 0.4979340 0.500
#> 33 poor_er 4 ROC 0.1046000 0.03125 0.4184000 0.125
#> 34 poor_er 4 PRC 0.2226168 0.12500 0.8904670 0.500
#> 35 good_er 4 ROC 0.1214000 0.03125 0.4856000 0.125
#> 36 good_er 4 PRC 0.2496078 0.12500 0.9984312 0.500
#> 37 excel 4 ROC 0.2337000 0.03125 0.9348000 0.125
#> 38 excel 4 PRC 0.2500000 0.12500 1.0000000 0.500
#> 39 perf 4 ROC 0.2500000 0.03125 1.0000000 0.125
#> 40 perf 4 PRC 0.2500000 0.12500 1.0000000 0.500
##################################################
### The McClish correction
###
## The pAUC rescaled between chance and perfect, as pROC reports it
pauc(sscurves.part, corrected = TRUE)
#> modnames dsids curvetypes paucs baselines spaucs sbaselines cpaucs
#> 1 m1 1 ROC 0.3771875 0.25 0.7543750 0.5 0.754375
#> 2 m1 1 PRC 0.3616708 0.25 0.7233417 0.5 NA