The auc_ci function takes an S3 object generated by
evalmod() and calculates CIs of AUCs when multiple data sets
are specified.
Usage
auc_ci(curves, alpha = NULL, dtype = NULL)
# S3 method for class 'aucs'
auc_ci(curves, alpha = 0.05, dtype = "normal")
# S3 method for class 'aucboot'
auc_ci(curves, alpha = 0.05, dtype = NULL)
# S3 method for class 'aucdelong'
auc_ci(curves, alpha = 0.05, dtype = NULL)Arguments
- curves
An
S3object generated byevalmod(). Theauc_cifunction accepts the following S3 objects.S3object# of models # of test datasets smcurves single multiple mmcurves multiple multiple See the Value section of
evalmod()for more details.It also accepts the two objects that describe the uncertainty of a single test set: an
aucbootobject fromauc_boot(), which gives a percentile interval, and anaucdelongobject fromauc_delong(), which gives a normal interval around DeLong's analytic standard error.- alpha
A numeric value of the significant level (default: 0.05)
- dtype
A string to specify the distribution used for CI calculation.
dtype distribution normal (default) Normal distribution z Normal distribution t t-distribution
Value
The auc_ci function returns a dataframe of AUC CIs, with a
baselines column beside the area giving what that area would be by
chance - 0.5 for a ROC curve and the proportion of positives for a
precision-recall curve. See Reading an area against its baseline in
auc().
Over several test datasets the baseline is averaged over the same
datasets the mean area is, so a fold that could not be evaluated is
left out of both. From auc_boot() or auc_delong() it is the
balance of the single test set.
An interval is the point of this function: the prevalence is what an
area is worth by chance in the limit, and an area from a finite sample
scatters around it, so an area sitting above its baseline means little
on its own. Read the baseline against the interval instead - see
The baseline is an asymptote in auc().
See also
evalmod() for generating S3 objects with
performance evaluation metrics. auc() for retrieving a dataset
of AUCs. auc_boot() and auc_delong() for a single test set.
Examples
##################################################
### Single model & multiple test datasets
###
## Create sample datasets with 100 positives and 100 negatives
samps <- create_sim_samples(4, 100, 100, "good_er")
mdat <- mmdata(samps[["scores"]], samps[["labels"]],
modnames = samps[["modnames"]],
dsids = samps[["dsids"]]
)
## Generate an smcurve object that contains ROC and Precision-Recall curves
smcurves <- evalmod(mdat)
## Calculate CI of AUCs
sm_auc_cis <- auc_ci(smcurves)
## Shows the result
sm_auc_cis
#> modnames curvetypes mean baselines error lower_bound upper_bound n
#> 1 good_er ROC 0.7950250 0.5 0.02369793 0.7713271 0.8187229 4
#> 2 good_er PRC 0.8343834 0.5 0.02143124 0.8129522 0.8558146 4
##################################################
### Multiple models & multiple test datasets
###
## Create sample datasets with 100 positives and 100 negatives
samps <- create_sim_samples(4, 100, 100, "all")
mdat <- mmdata(samps[["scores"]], samps[["labels"]],
modnames = samps[["modnames"]],
dsids = samps[["dsids"]]
)
## Generate an mscurve object that contains ROC and Precision-Recall curves
mmcurves <- evalmod(mdat)
## Calculate CI of AUCs
mm_auc_ci <- auc_ci(mmcurves)
## Shows the result
mm_auc_ci
#> modnames curvetypes mean baselines error lower_bound upper_bound
#> 1 random ROC 0.4925000 0.5 0.043150452 0.4493495 0.5356505
#> 2 random PRC 0.5105290 0.5 0.042359589 0.4681694 0.5528886
#> 3 poor_er ROC 0.7600750 0.5 0.027130623 0.7329444 0.7872056
#> 4 poor_er PRC 0.7132584 0.5 0.050140420 0.6631180 0.7633988
#> 5 good_er ROC 0.7716000 0.5 0.038758835 0.7328412 0.8103588
#> 6 good_er PRC 0.8094505 0.5 0.033099896 0.7763506 0.8425504
#> 7 excel ROC 0.9839500 0.5 0.009145040 0.9748050 0.9930950
#> 8 excel PRC 0.9844909 0.5 0.008032773 0.9764581 0.9925237
#> 9 perf ROC 1.0000000 0.5 0.000000000 1.0000000 1.0000000
#> 10 perf PRC 1.0000000 0.5 0.000000000 1.0000000 1.0000000
#> n
#> 1 4
#> 2 4
#> 3 4
#> 4 4
#> 5 4
#> 6 4
#> 7 4
#> 8 4
#> 9 4
#> 10 4