The auc_ci
function takes an S3
object generated by
evalmod
and calculates CIs of AUCs when multiple data sets
are specified.
auc_ci(curves, alpha = NULL, dtype = NULL)
# S3 method for class 'aucs'
auc_ci(curves, alpha = 0.05, dtype = "normal")
An S3
object generated by evalmod
.
The auc_ci
function accepts the following S3 objects.
S3 object | # of models | # of test datasets |
smcurves | single | multiple |
mmcurves | multiple | multiple |
See the Value section of evalmod
for more details.
A numeric value of the significant level (default: 0.05)
A string to specify the distribution used for CI calculation.
dtype | distribution |
normal (default) | Normal distribution |
z | Normal distribution |
t | t-distribution |
The auc_ci
function returns a dataframe of AUC CIs.
##################################################
### Single model & multiple test datasets
###
## Create sample datasets with 100 positives and 100 negatives
samps <- create_sim_samples(4, 100, 100, "good_er")
mdat <- mmdata(samps[["scores"]], samps[["labels"]],
modnames = samps[["modnames"]],
dsids = samps[["dsids"]]
)
## Generate an smcurve object that contains ROC and Precision-Recall curves
smcurves <- evalmod(mdat)
## Calculate CI of AUCs
sm_auc_cis <- auc_ci(smcurves)
## Shows the result
sm_auc_cis
#> modnames curvetypes mean error lower_bound upper_bound n
#> 1 good_er ROC 0.789975 0.02019134 0.7697837 0.8101663 4
#> 2 good_er PRC 0.834768 0.02176032 0.8130076 0.8565283 4
##################################################
### Multiple models & multiple test datasets
###
## Create sample datasets with 100 positives and 100 negatives
samps <- create_sim_samples(4, 100, 100, "all")
mdat <- mmdata(samps[["scores"]], samps[["labels"]],
modnames = samps[["modnames"]],
dsids = samps[["dsids"]]
)
## Generate an mscurve object that contains ROC and Precision-Recall curves
mmcurves <- evalmod(mdat)
## Calculate CI of AUCs
mm_auc_ci <- auc_ci(mmcurves)
## Shows the result
mm_auc_ci
#> modnames curvetypes mean error lower_bound upper_bound n
#> 1 random ROC 0.4785750 0.025405848 0.4531692 0.5039808 4
#> 2 random PRC 0.4927332 0.022360158 0.4703731 0.5150934 4
#> 3 poor_er ROC 0.7893750 0.043471334 0.7459037 0.8328463 4
#> 4 poor_er PRC 0.7458060 0.037821227 0.7079848 0.7836272 4
#> 5 good_er ROC 0.7734750 0.041057727 0.7324173 0.8145327 4
#> 6 good_er PRC 0.8099094 0.033761476 0.7761479 0.8436708 4
#> 7 excel ROC 0.9810500 0.006008337 0.9750417 0.9870583 4
#> 8 excel PRC 0.9813123 0.004354320 0.9769579 0.9856666 4
#> 9 perf ROC 1.0000000 0.000000000 1.0000000 1.0000000 4
#> 10 perf PRC 1.0000000 0.000000000 1.0000000 1.0000000 4