Calculate CIs of ROC and precision-recall AUCs

Source: R/g_auc_ci.R

The auc_ci function takes an S3 object generated by evalmod and calculates CIs of AUCs when multiple data sets are specified.

auc_ci(curves, alpha = NULL, dtype = NULL)

# S3 method for aucs
auc_ci(curves, alpha = 0.05, dtype = "normal")

Arguments

curves

An S3 object generated by evalmod. The auc_ci function accepts the following S3 objects.

S3 object# of models# of test datasets
smcurvessinglemultiple
mmcurvesmultiplemultiple

See the Value section of evalmod for more details.

alpha

A numeric value of the significant level (default: 0.05)

dtype

A string to specify the distribution used for CI calculation.

dtypedistribution
normal (default)Normal distribution
zNormal distribution
tt-distribution

Value

The auc_ci function returns a dataframe of AUC CIs.

See also

evalmod for generating S3 objects with performance evaluation measures. auc for retrieving a dataset of AUCs.

Examples


##################################################
### Single model & multiple test datasets
###

## Create sample datasets with 100 positives and 100 negatives
samps <- create_sim_samples(4, 100, 100, "good_er")
mdat <- mmdata(samps[["scores"]], samps[["labels"]],
  modnames = samps[["modnames"]],
  dsids = samps[["dsids"]]
)

## Generate an smcurve object that contains ROC and Precision-Recall curves
smcurves <- evalmod(mdat)

## Calculate CI of AUCs
sm_auc_cis <- auc_ci(smcurves)

## Shows the result
sm_auc_cis
#>   modnames curvetypes     mean      error lower_bound upper_bound n
#> 1  good_er        ROC 0.803000 0.03527781   0.7677222   0.8382778 4
#> 2  good_er        PRC 0.839866 0.02171738   0.8181486   0.8615834 4

##################################################
### Multiple models & multiple test datasets
###

## Create sample datasets with 100 positives and 100 negatives
samps <- create_sim_samples(4, 100, 100, "all")
mdat <- mmdata(samps[["scores"]], samps[["labels"]],
  modnames = samps[["modnames"]],
  dsids = samps[["dsids"]]
)

## Generate an mscurve object that contains ROC and Precision-Recall curves
mmcurves <- evalmod(mdat)

## Calculate CI of AUCs
mm_auc_ci <- auc_ci(mmcurves)

## Shows the result
mm_auc_ci
#>    modnames curvetypes      mean        error lower_bound upper_bound n
#> 1    random        ROC 0.4964000 0.0181331097   0.4782669   0.5145331 4
#> 2    random        PRC 0.4995228 0.0256121857   0.4739106   0.5251350 4
#> 3   poor_er        ROC 0.8077000 0.0206049613   0.7870950   0.8283050 4
#> 4   poor_er        PRC 0.7550536 0.0220631502   0.7329905   0.7771168 4
#> 5   good_er        ROC 0.8117500 0.0093396963   0.8024103   0.8210897 4
#> 6   good_er        PRC 0.8443380 0.0106722320   0.8336658   0.8550103 4
#> 7     excel        ROC 0.9823250 0.0045456624   0.9777793   0.9868707 4
#> 8     excel        PRC 0.9822201 0.0008885572   0.9813316   0.9831087 4
#> 9      perf        ROC 1.0000000 0.0000000000   1.0000000   1.0000000 4
#> 10     perf        PRC 1.0000000 0.0000000000   1.0000000   1.0000000 4