The evalmod function calculates ROC and Precision-Recall curves for
specified prediction scores and binary labels. It also calculate several
basic performance evaluation measures, such as accuracy, error rate, and
precision, by specifying mode as "basic".
evalmod(
mdat,
mode = NULL,
scores = NULL,
labels = NULL,
modnames = NULL,
dsids = NULL,
posclass = NULL,
na_worst = TRUE,
ties_method = "equiv",
calc_avg = TRUE,
cb_alpha = 0.05,
raw_curves = FALSE,
x_bins = 1000,
interpolate = TRUE,
...
)An S3 object created by the mmdata
function. It contains formatted scores and labels.
The evalmod function ignores the following arguments
when mdat is specified.
scores
labels
modnames
dsids
posclass
na_worst
ties_method
These arguments are internally passed to the mmdata function
when mdat is unspecified.
In that case, both scores and labels must be
at least specified.
A string that specifies the types of evaluation measures
that the evalmod function calculates.
ROC and Precision-Recall curves
Same as above
Normalized ranks vs. accuracy, error rate, specificity, sensitivity, precision, Matthews correlation coefficient, and F-score.
Fast AUC(ROC) calculation with the U statistic
A numeric dataset of predicted scores. It can be a vector,
a matrix, an array, a data frame, or a list. The join_scores
function can be useful to make scores with multiple datasets.
A numeric, character, logical, or factor dataset
of observed labels. It can be a vector, a matrix, an array,
a data frame, or a list. The join_labels
function can be useful to make labels with multiple datasets.
A character vector for the names of the models.
The evalmod function automatically generates default names
as "m1", "m2", "m3", and so on when it is NULL.
A numeric vector for test dataset IDs.
The evalmod function automatically generates the default ID
as 1 when it is NULL.
A scalar value to specify the label of positives
in labels. It must be the same data type as labels.
For example, posclass = -1 changes the positive label
from 1 to -1 when labels contains
1 and -1. The positive label will be automatically
detected when posclass is NULL.
A Boolean value for controlling the treatment of NAs
in scores.
All NAs are treated as the worst scores
All NAs are treated as the best scores
A string for controlling ties in scores.
Ties are equivalently ranked
Ties are ranked in an increasing order as appeared
Ties are ranked in random order
A logical value to specify whether average curves should
be calculated. It is effective only when dsids contains multiple
dataset IDs. For instance, the function calculates the average for the
model "m1" when modnames is c("m1", "m1", "m1") and
dsids is c(1, 2, 3). The calculation points are defined by
x_bins.
A numeric value with range [0, 1] to specify the alpha
value of the point-wise confidence bounds calculation. It is effective only
when calc_avg is set to TRUE. For example, it should be
0.05 for the 95% confidence level. The calculation points are
defined by x_bins.
A logical value to specify whether all raw curves
should be discarded after the average curves are calculated.
It is effective only when calc_avg is set to TRUE.
An integer value to specify the number of minimum bins
on the x-axis. It is then used to define supporting points For instance,
the x-values of the supporting points will be c(0, 0.5, 1) and
c(0, 0.25, 0.5, 0.75, 1) when x_bins = 2
and x_bins = 4, respectively. All corresponding y-values of
the supporting points are calculated. x_bins is effective only
when mode is set to rocprc or prcroc.
A Boolean value to specify whether or not
interpolation of ROC and precision-recall curves are
performed. x_bins and calc_avg are
ignored and when x_bins is set to FALSE.
interpolate is effective only when mode is set
to rocprc or prcroc.
These additional arguments are passed to mmdata
for data preparation.
The evalmod function returns an S3 object
that contains performance evaluation measures. The number of models and
the number of datasets can be controlled by modnames and
dsids. For example, the number of models is "single" and the number
of test datasets is "multiple" when modnames = c("m1", "m1", "m1")
and dsids = c(1, 2, 3) are specified.
Different S3 objects have different default behaviors of S3
generics, such as plot, autoplot, and
The evalmod function returns one of the following S3
objects when mode is "prcroc".
The objects contain ROC and Precision-Recall curves.
S3 object | # of models | # of test datasets |
| sscurves | single | single |
| mscurves | multiple | single |
| smcurves | single | multiple |
| mmcurves | multiple | multiple |
The evalmod function returns one of the following S3
objects when mode is "basic".
They contain five different basic evaluation measures; error rate,
accuracy, specificity, sensitivity, and precision.
S3 object | # of models | # of test datasets |
| sspoints | single | single |
| mspoints | multiple | single |
| smpoints | single | multiple |
| mmpoints | multiple | multiple |
The evalmod function returns the aucroc S3 object
when mode is "aucroc", which can be used with 'print'
and 'as.data.frame'.
plot for plotting curves with the general R plot.
autoplot and fortify for plotting curves
with ggplot2. mmdata for formatting input data.
join_scores and join_labels for formatting
scores and labels with multiple datasets.
format_nfold for creating n-fold cross validation dataset
from data frame.
create_sim_samples for generating random samples
for simulations.
##################################################
### Single model & single test dataset
###
## Load a dataset with 10 positives and 10 negatives
data(P10N10)
## Generate an sscurve object that contains ROC and Precision-Recall curves
sscurves <- evalmod(scores = P10N10$scores, labels = P10N10$labels)
sscurves
#>
#> === AUCs ===
#>
#> Model name Dataset ID Curve type AUC
#> 1 m1 1 ROC 0.7200000
#> 2 m1 1 PRC 0.7397716
#>
#>
#> === Input data ===
#>
#> Model name Dataset ID # of negatives # of positives
#> 1 m1 1 10 10
#>
## Generate an sspoints object that contains basic evaluation measures
sspoints <- evalmod(
mode = "basic", scores = P10N10$scores,
labels = P10N10$labels
)
sspoints
#>
#> === Basic performance evaluation measures ===
#>
#> ## Performance measures (Meas.)
#> rank: normalized rank
#> score: score
#> label: label
#> err: error rate
#> acc: accuracy
#> sp: specificity
#> sn: sensitivity
#> prec: precision
#> mcc: Matthews correlation coefficient
#> fscore: F-score
#>
#>
#> Model ID Meas. Min. 1st Qu. Median Mean 3rd Qu.
#> 1 m1 1 rank 0.0000000 0.2500000 0.5000000 0.5000000 0.7500000
#> 2 m1 1 score 5.0000000 5.7500000 14.0000000 11.7500000 15.2500000
#> 3 m1 1 label -1.0000000 -1.0000000 0.0000000 0.0000000 1.0000000
#> 4 m1 1 err 0.3000000 0.3500000 0.4000000 0.3952381 0.4400000
#> 5 m1 1 acc 0.5000000 0.5600000 0.6000000 0.6047619 0.6500000
#> 6 m1 1 sp 0.0000000 0.4000000 0.6333333 0.6047619 0.9000000
#> 7 m1 1 sn 0.0000000 0.4000000 0.6333333 0.6047619 0.9000000
#> 8 m1 1 prec 0.5000000 0.5750000 0.6333333 0.6892147 0.7619048
#> 9 m1 1 mcc 0.1376494 0.2238168 0.2666667 0.2755698 0.3367701
#> 10 m1 1 fscore 0.0000000 0.5333333 0.6333333 0.5579798 0.6758621
#> Max.
#> 1 1.0000000
#> 2 20.0000000
#> 3 1.0000000
#> 4 0.5000000
#> 5 0.7000000
#> 6 1.0000000
#> 7 1.0000000
#> 8 1.0000000
#> 9 0.4364358
#> 10 0.7200000
#>
#>
#> === Input data ===
#>
#> Model name Dataset ID # of negatives # of positives
#> 1 m1 1 10 10
#>
##################################################
### Multiple models & single test dataset
###
## Create sample datasets with 100 positives and 100 negatives
samps <- create_sim_samples(1, 100, 100, "all")
mdat <- mmdata(samps[["scores"]], samps[["labels"]],
modnames = samps[["modnames"]]
)
## Generate an mscurve object that contains ROC and Precision-Recall curves
mscurves <- evalmod(mdat)
mscurves
#>
#> === AUCs ===
#>
#> Model name Dataset ID Curve type AUC
#> 1 random 1 ROC 0.5158000
#> 2 random 1 PRC 0.4985065
#> 3 poor_er 1 ROC 0.8139000
#> 4 poor_er 1 PRC 0.7898648
#> 5 good_er 1 ROC 0.7675000
#> 6 good_er 1 PRC 0.8027195
#> 7 excel 1 ROC 0.9897000
#> 8 excel 1 PRC 0.9904196
#> 9 perf 1 ROC 1.0000000
#> 10 perf 1 PRC 1.0000000
#>
#>
#> === Input data ===
#>
#> Model name Dataset ID # of negatives # of positives
#> 1 random 1 100 100
#> 2 poor_er 1 100 100
#> 3 good_er 1 100 100
#> 4 excel 1 100 100
#> 5 perf 1 100 100
#>
## Generate an mspoints object that contains basic evaluation measures
mspoints <- evalmod(mdat, mode = "basic")
mspoints
#>
#> === Basic performance evaluation measures ===
#>
#> ## Performance measures (Meas.)
#> rank: normalized rank
#> score: score
#> label: label
#> err: error rate
#> acc: accuracy
#> sp: specificity
#> sn: sensitivity
#> prec: precision
#> mcc: Matthews correlation coefficient
#> fscore: F-score
#>
#>
#> Model ID Meas. Min. 1st Qu. Median Mean
#> 1 random 1 rank 0.000000000 0.25000000 0.50000000 0.50000000
#> 2 random 1 score -3.677000017 -0.69286625 -0.01046012 -0.03022486
#> 3 random 1 label -1.000000000 -1.00000000 0.00000000 0.00000000
#> 4 random 1 err 0.450000000 0.48500000 0.49000000 0.49213930
#> 5 random 1 acc 0.465000000 0.50000000 0.51000000 0.50786070
#> 6 random 1 sp 0.000000000 0.26000000 0.52000000 0.50786070
#> 7 random 1 sn 0.000000000 0.25000000 0.52000000 0.50786070
#> 8 random 1 prec 0.166666667 0.50000000 0.50793651 0.49507401
#> 9 random 1 mcc -0.117242076 0.00000000 0.02060214 0.01582692
#> 10 random 1 fscore 0.000000000 0.33557047 0.51741294 0.45531305
#> 11 poor_er 1 rank 0.000000000 0.25000000 0.50000000 0.50000000
#> 12 poor_er 1 score 0.004556939 0.45334996 0.71025134 0.64337548
#> 13 poor_er 1 label -1.000000000 -1.00000000 0.00000000 0.00000000
#> 14 poor_er 1 err 0.235000000 0.29500000 0.32500000 0.34383085
#> 15 poor_er 1 acc 0.500000000 0.61000000 0.67500000 0.65616915
#> 16 poor_er 1 sp 0.000000000 0.45000000 0.71000000 0.65616915
#> 17 poor_er 1 sn 0.000000000 0.41000000 0.71000000 0.65616915
#> 18 poor_er 1 prec 0.500000000 0.63087248 0.71428571 0.72504871
#> 19 poor_er 1 mcc 0.070888121 0.32530269 0.38756664 0.37347769
#> 20 poor_er 1 fscore 0.000000000 0.54666667 0.68493151 0.61138545
#> 21 good_er 1 rank 0.000000000 0.25000000 0.50000000 0.50000000
#> 22 good_er 1 score 0.001641075 0.09601156 0.26430949 0.32978000
#> 23 good_er 1 label -1.000000000 -1.00000000 0.00000000 0.00000000
#> 24 good_er 1 err 0.285000000 0.33000000 0.35000000 0.36691542
#> 25 good_er 1 acc 0.500000000 0.59500000 0.65000000 0.63308458
#> 26 good_er 1 sp 0.000000000 0.41000000 0.67000000 0.63308458
#> 27 good_er 1 sn 0.000000000 0.42000000 0.67000000 0.63308458
#> 28 good_er 1 prec 0.500000000 0.59712230 0.66666667 0.72546197
#> 29 good_er 1 mcc 0.070888121 0.28966647 0.33282435 0.32190780
#> 30 good_er 1 fscore 0.000000000 0.55844156 0.67114094 0.59561262
#> 31 excel 1 rank 0.000000000 0.25000000 0.50000000 0.50000000
#> 32 excel 1 score -2.659147186 0.01873824 1.51734061 1.58446194
#> 33 excel 1 label -1.000000000 -1.00000000 0.00000000 0.00000000
#> 34 excel 1 err 0.045000000 0.12500000 0.25000000 0.25636816
#> 35 excel 1 acc 0.500000000 0.62500000 0.75000000 0.74363184
#> 36 excel 1 sp 0.000000000 0.50000000 0.93000000 0.74363184
#> 37 excel 1 sn 0.000000000 0.50000000 0.93000000 0.74363184
#> 38 excel 1 prec 0.500000000 0.66666667 0.92929293 0.84105350
#> 39 excel 1 mcc 0.070888121 0.38226007 0.57735027 0.56314557
#> 40 excel 1 fscore 0.000000000 0.66666667 0.76335878 0.70535011
#> 41 perf 1 rank 0.000000000 0.25000000 0.50000000 0.50000000
#> 42 perf 1 score 0.000000000 0.00000000 0.50000000 0.50000000
#> 43 perf 1 label -1.000000000 -1.00000000 0.00000000 0.00000000
#> 44 perf 1 err 0.000000000 0.12500000 0.25000000 0.25124378
#> 45 perf 1 acc 0.500000000 0.62500000 0.75000000 0.74875622
#> 46 perf 1 sp 0.000000000 0.50000000 1.00000000 0.74875622
#> 47 perf 1 sn 0.000000000 0.50000000 1.00000000 0.74875622
#> 48 perf 1 prec 0.500000000 0.66666667 1.00000000 0.84609623
#> 49 perf 1 mcc 0.070888121 0.38226007 0.57735027 0.57352524
#> 50 perf 1 fscore 0.000000000 0.66666667 0.76335878 0.71042736
#> 3rd Qu. Max.
#> 1 0.75000000 1.0000000
#> 2 0.67744431 2.5727954
#> 3 1.00000000 1.0000000
#> 4 0.50000000 0.5350000
#> 5 0.51500000 0.5500000
#> 6 0.75000000 1.0000000
#> 7 0.76000000 1.0000000
#> 8 0.51666667 1.0000000
#> 9 0.04010041 0.1400280
#> 10 0.60728745 0.6689420
#> 11 0.75000000 1.0000000
#> 12 0.87509067 0.9956432
#> 13 1.00000000 1.0000000
#> 14 0.39000000 0.5000000
#> 15 0.70500000 0.7650000
#> 16 0.91000000 1.0000000
#> 17 0.95000000 1.0000000
#> 18 0.82000000 1.0000000
#> 19 0.45062442 0.5473816
#> 20 0.75098814 0.7911111
#> 21 0.75000000 1.0000000
#> 22 0.48688719 0.9851729
#> 23 1.00000000 1.0000000
#> 24 0.40500000 0.5000000
#> 25 0.67000000 0.7150000
#> 26 0.92000000 1.0000000
#> 27 0.91000000 1.0000000
#> 28 0.84000000 1.0000000
#> 29 0.39390284 0.4506853
#> 30 0.69372694 0.7330677
#> 31 0.75000000 1.0000000
#> 32 3.17711427 5.4639346
#> 33 1.00000000 1.0000000
#> 34 0.37500000 0.5000000
#> 35 0.87500000 0.9550000
#> 36 1.00000000 1.0000000
#> 37 1.00000000 1.0000000
#> 38 1.00000000 1.0000000
#> 39 0.77459667 0.9122377
#> 40 0.87640449 0.9533679
#> 41 0.75000000 1.0000000
#> 42 1.00000000 1.0000000
#> 43 1.00000000 1.0000000
#> 44 0.37500000 0.5000000
#> 45 0.87500000 1.0000000
#> 46 1.00000000 1.0000000
#> 47 1.00000000 1.0000000
#> 48 1.00000000 1.0000000
#> 49 0.77459667 1.0000000
#> 50 0.87640449 1.0000000
#>
#>
#> === Input data ===
#>
#> Model name Dataset ID # of negatives # of positives
#> 1 random 1 100 100
#> 2 poor_er 1 100 100
#> 3 good_er 1 100 100
#> 4 excel 1 100 100
#> 5 perf 1 100 100
#>
##################################################
### Single model & multiple test datasets
###
## Create sample datasets with 100 positives and 100 negatives
samps <- create_sim_samples(4, 100, 100, "good_er")
mdat <- mmdata(samps[["scores"]], samps[["labels"]],
modnames = samps[["modnames"]],
dsids = samps[["dsids"]]
)
## Generate an smcurve object that contains ROC and Precision-Recall curves
smcurves <- evalmod(mdat)
smcurves
#>
#> === AUCs ===
#>
#> Model name Dataset ID Curve type AUC
#> 1 good_er 1 ROC 0.8182000
#> 2 good_er 1 PRC 0.8354029
#> 3 good_er 2 ROC 0.8075000
#> 4 good_er 2 PRC 0.8480362
#> 5 good_er 3 ROC 0.7865000
#> 6 good_er 3 PRC 0.8281439
#> 7 good_er 4 ROC 0.8556000
#> 8 good_er 4 PRC 0.8755816
#>
#>
#> === Input data ===
#>
#> Model name Dataset ID # of negatives # of positives
#> 1 good_er 1 100 100
#> 2 good_er 2 100 100
#> 3 good_er 3 100 100
#> 4 good_er 4 100 100
#>
## Generate an smpoints object that contains basic evaluation measures
smpoints <- evalmod(mdat, mode = "basic")
smpoints
#>
#> === Basic performance evaluation measures ===
#>
#> ## Performance measures (Meas.)
#> rank: normalized rank
#> score: score
#> label: label
#> err: error rate
#> acc: accuracy
#> sp: specificity
#> sn: sensitivity
#> prec: precision
#> mcc: Matthews correlation coefficient
#> fscore: F-score
#>
#>
#> Model ID Meas. Min. 1st Qu. Median Mean 3rd Qu.
#> 1 good_er 1 rank 0.0000000000 0.25000000 0.5000000 0.5000000 0.7500000
#> 2 good_er 1 score 0.0027372587 0.08630857 0.2554727 0.3364887 0.5583914
#> 3 good_er 1 label -1.0000000000 -1.00000000 0.0000000 0.0000000 1.0000000
#> 4 good_er 1 err 0.2400000000 0.27000000 0.3250000 0.3416915 0.4000000
#> 5 good_er 1 acc 0.5000000000 0.60000000 0.6750000 0.6583085 0.7300000
#> 6 good_er 1 sp 0.0000000000 0.40000000 0.7400000 0.6583085 0.9500000
#> 7 good_er 1 sn 0.0000000000 0.45000000 0.7400000 0.6583085 0.9000000
#> 8 good_er 1 prec 0.5000000000 0.60000000 0.7475728 0.7489318 0.8852459
#> 9 good_er 1 mcc 0.0708881205 0.28791823 0.4008919 0.3758310 0.4803597
#> 10 good_er 1 fscore 0.0000000000 0.59602649 0.6975089 0.6201332 0.7317073
#> 11 good_er 2 rank 0.0000000000 0.25000000 0.5000000 0.5000000 0.7500000
#> 12 good_er 2 score 0.0018669488 0.10025507 0.2737964 0.3413153 0.5290876
#> 13 good_er 2 label -1.0000000000 -1.00000000 0.0000000 0.0000000 1.0000000
#> 14 good_er 2 err 0.2350000000 0.27500000 0.3350000 0.3470149 0.4050000
#> 15 good_er 2 acc 0.5000000000 0.59500000 0.6650000 0.6529851 0.7250000
#> 16 good_er 2 sp 0.0000000000 0.38000000 0.7300000 0.6529851 0.9600000
#> 17 good_er 2 sn 0.0000000000 0.46000000 0.7300000 0.6529851 0.8800000
#> 18 good_er 2 prec 0.5000000000 0.58552632 0.7300000 0.7533973 0.9215686
#> 19 good_er 2 mcc 0.0000000000 0.27141905 0.3821344 0.3612264 0.4722113
#> 20 good_er 2 fscore 0.0000000000 0.61744966 0.6953125 0.6184720 0.7168142
#> 21 good_er 3 rank 0.0000000000 0.25000000 0.5000000 0.5000000 0.7500000
#> 22 good_er 3 score 0.0005176733 0.14608422 0.3114677 0.3570825 0.5458316
#> 23 good_er 3 label -1.0000000000 -1.00000000 0.0000000 0.0000000 1.0000000
#> 24 good_er 3 err 0.2500000000 0.30000000 0.3350000 0.3574627 0.4200000
#> 25 good_er 3 acc 0.4950000000 0.58000000 0.6650000 0.6425373 0.7000000
#> 26 good_er 3 sp 0.0000000000 0.35000000 0.7100000 0.6425373 0.9400000
#> 27 good_er 3 sn 0.0000000000 0.44000000 0.7100000 0.6425373 0.8500000
#> 28 good_er 3 prec 0.4974874372 0.56666667 0.7100000 0.7409599 0.8833333
#> 29 good_er 3 mcc -0.0708881205 0.22092416 0.3773825 0.3377050 0.4424347
#> 30 good_er 3 fscore 0.0000000000 0.58666667 0.6816479 0.6077819 0.7081340
#> 31 good_er 4 rank 0.0000000000 0.25000000 0.5000000 0.5000000 0.7500000
#> 32 good_er 4 score 0.0070266103 0.13068680 0.2946315 0.3675021 0.5561777
#> 33 good_er 4 label -1.0000000000 -1.00000000 0.0000000 0.0000000 1.0000000
#> 34 good_er 4 err 0.1900000000 0.24500000 0.3050000 0.3230846 0.3950000
#> 35 good_er 4 acc 0.5000000000 0.60500000 0.6950000 0.6769154 0.7550000
#> 36 good_er 4 sp 0.0000000000 0.44000000 0.7900000 0.6769154 0.9700000
#> 37 good_er 4 sn 0.0000000000 0.47000000 0.7900000 0.6769154 0.9400000
#> 38 good_er 4 prec 0.5000000000 0.62666667 0.7900000 0.7731264 0.9400000
#> 39 good_er 4 mcc 0.0708881205 0.32566062 0.4506853 0.4180102 0.5323186
#> 40 good_er 4 fscore 0.0000000000 0.62666667 0.7159091 0.6399487 0.7553648
#> Max.
#> 1 1.0000000
#> 2 0.9915508
#> 3 1.0000000
#> 4 0.5000000
#> 5 0.7600000
#> 6 1.0000000
#> 7 1.0000000
#> 8 1.0000000
#> 9 0.5209385
#> 10 0.7747748
#> 11 1.0000000
#> 12 0.9896095
#> 13 1.0000000
#> 14 0.5000000
#> 15 0.7650000
#> 16 1.0000000
#> 17 1.0000000
#> 18 1.0000000
#> 19 0.5405484
#> 20 0.7488152
#> 21 1.0000000
#> 22 0.9888999
#> 23 1.0000000
#> 24 0.5050000
#> 25 0.7500000
#> 26 1.0000000
#> 27 1.0000000
#> 28 1.0000000
#> 29 0.5144958
#> 30 0.7291667
#> 31 1.0000000
#> 32 0.9702179
#> 33 1.0000000
#> 34 0.5000000
#> 35 0.8100000
#> 36 1.0000000
#> 37 1.0000000
#> 38 1.0000000
#> 39 0.6211190
#> 40 0.8041237
#>
#>
#> === Input data ===
#>
#> Model name Dataset ID # of negatives # of positives
#> 1 good_er 1 100 100
#> 2 good_er 2 100 100
#> 3 good_er 3 100 100
#> 4 good_er 4 100 100
#>
##################################################
### Multiple models & multiple test datasets
###
## Create sample datasets with 100 positives and 100 negatives
samps <- create_sim_samples(4, 100, 100, "all")
mdat <- mmdata(samps[["scores"]], samps[["labels"]],
modnames = samps[["modnames"]],
dsids = samps[["dsids"]]
)
## Generate an mmcurve object that contains ROC and Precision-Recall curves
mmcurves <- evalmod(mdat)
mmcurves
#>
#> === AUCs ===
#>
#> Model name Dataset ID Curve type AUC
#> 1 random 1 ROC 0.4525000
#> 2 random 1 PRC 0.4478566
#> 3 poor_er 1 ROC 0.8364000
#> 4 poor_er 1 PRC 0.8094832
#> 5 good_er 1 ROC 0.8007000
#> 6 good_er 1 PRC 0.8414833
#> 7 excel 1 ROC 0.9803000
#> 8 excel 1 PRC 0.9838961
#> 9 perf 1 ROC 1.0000000
#> 10 perf 1 PRC 1.0000000
#> 11 random 2 ROC 0.5322000
#> 12 random 2 PRC 0.5687969
#> 13 poor_er 2 ROC 0.7845000
#> 14 poor_er 2 PRC 0.7313286
#> 15 good_er 2 ROC 0.8493000
#> 16 good_er 2 PRC 0.8651321
#> 17 excel 2 ROC 0.9937000
#> 18 excel 2 PRC 0.9939577
#> 19 perf 2 ROC 1.0000000
#> 20 perf 2 PRC 1.0000000
#> 21 random 3 ROC 0.4200000
#> 22 random 3 PRC 0.4470086
#> 23 poor_er 3 ROC 0.8446000
#> 24 poor_er 3 PRC 0.8037929
#> 25 good_er 3 ROC 0.8445000
#> 26 good_er 3 PRC 0.8734010
#> 27 excel 3 ROC 0.9927000
#> 28 excel 3 PRC 0.9933865
#> 29 perf 3 ROC 1.0000000
#> 30 perf 3 PRC 1.0000000
#> 31 random 4 ROC 0.5076000
#> 32 random 4 PRC 0.4877154
#> 33 poor_er 4 ROC 0.8011000
#> 34 poor_er 4 PRC 0.7376218
#> 35 good_er 4 ROC 0.8050000
#> 36 good_er 4 PRC 0.8345215
#> 37 excel 4 ROC 0.9853000
#> 38 excel 4 PRC 0.9865754
#> 39 perf 4 ROC 1.0000000
#> 40 perf 4 PRC 1.0000000
#>
#>
#> === Input data ===
#>
#> Model name Dataset ID # of negatives # of positives
#> 1 random 1 100 100
#> 2 poor_er 1 100 100
#> 3 good_er 1 100 100
#> 4 excel 1 100 100
#> 5 perf 1 100 100
#> 6 random 2 100 100
#> 7 poor_er 2 100 100
#> 8 good_er 2 100 100
#> 9 excel 2 100 100
#> 10 perf 2 100 100
#> 11 random 3 100 100
#> 12 poor_er 3 100 100
#> 13 good_er 3 100 100
#> 14 excel 3 100 100
#> 15 perf 3 100 100
#> 16 random 4 100 100
#> 17 poor_er 4 100 100
#> 18 good_er 4 100 100
#> 19 excel 4 100 100
#> 20 perf 4 100 100
#>
## Generate an mmpoints object that contains basic evaluation measures
mmpoints <- evalmod(mdat, mode = "basic")
mmpoints
#>
#> === Basic performance evaluation measures ===
#>
#> ## Performance measures (Meas.)
#> rank: normalized rank
#> score: score
#> label: label
#> err: error rate
#> acc: accuracy
#> sp: specificity
#> sn: sensitivity
#> prec: precision
#> mcc: Matthews correlation coefficient
#> fscore: F-score
#>
#>
#> Model ID Meas. Min. 1st Qu. Median Mean
#> 1 random 1 rank 0.0000000000 0.25000000 0.50000000 0.500000000
#> 2 random 1 score -2.7645393141 -0.67601821 -0.04667093 -0.017915600
#> 3 random 1 label -1.0000000000 -1.00000000 0.00000000 0.000000000
#> 4 random 1 err 0.4600000000 0.50000000 0.53000000 0.523631841
#> 5 random 1 acc 0.4200000000 0.45000000 0.47000000 0.476368159
#> 6 random 1 sp 0.0000000000 0.28000000 0.46000000 0.476368159
#> 7 random 1 sn 0.0000000000 0.19000000 0.46000000 0.476368159
#> 8 random 1 prec 0.0000000000 0.38461538 0.45833333 0.436957703
#> 9 random 1 mcc -0.1781741613 -0.11174544 -0.07235746 -0.057008636
#> 10 random 1 fscore 0.0000000000 0.25165563 0.46231156 0.419204893
#> 11 poor_er 1 rank 0.0000000000 0.25000000 0.50000000 0.500000000
#> 12 poor_er 1 score 0.0363403885 0.46404930 0.73503693 0.647283931
#> 13 poor_er 1 label -1.0000000000 -1.00000000 0.00000000 0.000000000
#> 14 poor_er 1 err 0.2200000000 0.26500000 0.30000000 0.332636816
#> 15 poor_er 1 acc 0.5000000000 0.60500000 0.70000000 0.667363184
#> 16 poor_er 1 sp 0.0000000000 0.45000000 0.77000000 0.667363184
#> 17 poor_er 1 sn 0.0000000000 0.41000000 0.77000000 0.667363184
#> 18 poor_er 1 prec 0.5000000000 0.63870968 0.76530612 0.737683899
#> 19 poor_er 1 mcc 0.0708881205 0.32566062 0.44739012 0.397095975
#> 20 poor_er 1 fscore 0.0000000000 0.54666667 0.71428571 0.623001258
#> 21 good_er 1 rank 0.0000000000 0.25000000 0.50000000 0.500000000
#> 22 good_er 1 score 0.0034963966 0.13536600 0.32474063 0.375668403
#> 23 good_er 1 label -1.0000000000 -1.00000000 0.00000000 0.000000000
#> 24 good_er 1 err 0.2400000000 0.29500000 0.33500000 0.350398010
#> 25 good_er 1 acc 0.5000000000 0.60000000 0.66500000 0.649601990
#> 26 good_er 1 sp 0.0000000000 0.40000000 0.69000000 0.649601990
#> 27 good_er 1 sn 0.0000000000 0.47000000 0.69000000 0.649601990
#> 28 good_er 1 prec 0.5000000000 0.60000000 0.69000000 0.749145308
#> 29 good_er 1 mcc 0.0000000000 0.29019428 0.36181361 0.356013013
#> 30 good_er 1 fscore 0.0000000000 0.62666667 0.69064748 0.614530493
#> 31 excel 1 rank 0.0000000000 0.25000000 0.50000000 0.500000000
#> 32 excel 1 score -2.2050629700 -0.18231032 1.75134134 1.492438207
#> 33 excel 1 label -1.0000000000 -1.00000000 0.00000000 0.000000000
#> 34 excel 1 err 0.0650000000 0.13000000 0.25500000 0.261044776
#> 35 excel 1 acc 0.5000000000 0.62500000 0.74500000 0.738955224
#> 36 excel 1 sp 0.0000000000 0.49000000 0.92000000 0.738955224
#> 37 excel 1 sn 0.0000000000 0.50000000 0.92000000 0.738955224
#> 38 excel 1 prec 0.5000000000 0.66000000 0.92000000 0.837121386
#> 39 excel 1 mcc 0.0708881205 0.37630912 0.56965192 0.552760567
#> 40 excel 1 fscore 0.0000000000 0.66666667 0.75776398 0.701138644
#> 41 perf 1 rank 0.0000000000 0.25000000 0.50000000 0.500000000
#> 42 perf 1 score 0.0000000000 0.00000000 0.50000000 0.500000000
#> 43 perf 1 label -1.0000000000 -1.00000000 0.00000000 0.000000000
#> 44 perf 1 err 0.0000000000 0.12500000 0.25000000 0.251243781
#> 45 perf 1 acc 0.5000000000 0.62500000 0.75000000 0.748756219
#> 46 perf 1 sp 0.0000000000 0.50000000 1.00000000 0.748756219
#> 47 perf 1 sn 0.0000000000 0.50000000 1.00000000 0.748756219
#> 48 perf 1 prec 0.5000000000 0.66666667 1.00000000 0.846096234
#> 49 perf 1 mcc 0.0708881205 0.38226007 0.57735027 0.573525244
#> 50 perf 1 fscore 0.0000000000 0.66666667 0.76335878 0.710427365
#> 51 random 2 rank 0.0000000000 0.25000000 0.50000000 0.500000000
#> 52 random 2 score -2.2580580212 -0.61441622 -0.01016459 0.001346863
#> 53 random 2 label -1.0000000000 -1.00000000 0.00000000 0.000000000
#> 54 random 2 err 0.4350000000 0.47000000 0.48500000 0.483980100
#> 55 random 2 acc 0.4650000000 0.50000000 0.51500000 0.516019900
#> 56 random 2 sp 0.0000000000 0.25000000 0.50000000 0.516019900
#> 57 random 2 sn 0.0000000000 0.30000000 0.50000000 0.516019900
#> 58 random 2 prec 0.4787878788 0.50000000 0.52173913 0.556619606
#> 59 random 2 mcc -0.1263227882 0.00000000 0.04222003 0.037598301
#> 60 random 2 fscore 0.0000000000 0.40000000 0.50000000 0.471870692
#> 61 poor_er 2 rank 0.0000000000 0.25000000 0.50000000 0.500000000
#> 62 poor_er 2 score 0.0045880578 0.46147546 0.73349331 0.653208990
#> 63 poor_er 2 label -1.0000000000 -1.00000000 0.00000000 0.000000000
#> 64 poor_er 2 err 0.2650000000 0.30500000 0.33500000 0.358457711
#> 65 poor_er 2 acc 0.4950000000 0.59500000 0.66500000 0.641542289
#> 66 poor_er 2 sp 0.0000000000 0.41000000 0.71000000 0.641542289
#> 67 poor_er 2 sn 0.0000000000 0.42000000 0.71000000 0.641542289
#> 68 poor_er 2 prec 0.0000000000 0.59477124 0.68141593 0.681626141
#> 69 poor_er 2 mcc -0.0708881205 0.28163911 0.37500000 0.334544946
#> 70 poor_er 2 fscore 0.0000000000 0.56000000 0.70175439 0.596861226
#> 71 good_er 2 rank 0.0000000000 0.25000000 0.50000000 0.500000000
#> 72 good_er 2 score 0.0016156777 0.14294125 0.28658272 0.356380310
#> 73 good_er 2 label -1.0000000000 -1.00000000 0.00000000 0.000000000
#> 74 good_er 2 err 0.2000000000 0.24500000 0.31000000 0.326218905
#> 75 good_er 2 acc 0.5000000000 0.60000000 0.69000000 0.673781095
#> 76 good_er 2 sp 0.0000000000 0.43000000 0.79000000 0.673781095
#> 77 good_er 2 sn 0.0000000000 0.45000000 0.79000000 0.673781095
#> 78 good_er 2 prec 0.5000000000 0.62000000 0.79000000 0.767389969
#> 79 good_er 2 mcc 0.0708881205 0.30730550 0.44487826 0.407213315
#> 80 good_er 2 fscore 0.0000000000 0.60000000 0.69818182 0.636179006
#> 81 excel 2 rank 0.0000000000 0.25000000 0.50000000 0.500000000
#> 82 excel 2 score -2.8722643040 -0.01415298 1.41642237 1.484108160
#> 83 excel 2 label -1.0000000000 -1.00000000 0.00000000 0.000000000
#> 84 excel 2 err 0.0350000000 0.12500000 0.25000000 0.254378109
#> 85 excel 2 acc 0.5000000000 0.62500000 0.75000000 0.745621891
#> 86 excel 2 sp 0.0000000000 0.50000000 0.95000000 0.745621891
#> 87 excel 2 sn 0.0000000000 0.50000000 0.95000000 0.745621891
#> 88 excel 2 prec 0.5000000000 0.66666667 0.95145631 0.842968050
#> 89 excel 2 mcc 0.0708881205 0.38226007 0.57735027 0.567179576
#> 90 excel 2 fscore 0.0000000000 0.66666667 0.76335878 0.707299371
#> 91 perf 2 rank 0.0000000000 0.25000000 0.50000000 0.500000000
#> 92 perf 2 score 0.0000000000 0.00000000 0.50000000 0.500000000
#> 93 perf 2 label -1.0000000000 -1.00000000 0.00000000 0.000000000
#> 94 perf 2 err 0.0000000000 0.12500000 0.25000000 0.251243781
#> 95 perf 2 acc 0.5000000000 0.62500000 0.75000000 0.748756219
#> 96 perf 2 sp 0.0000000000 0.50000000 1.00000000 0.748756219
#> 97 perf 2 sn 0.0000000000 0.50000000 1.00000000 0.748756219
#> 98 perf 2 prec 0.5000000000 0.66666667 1.00000000 0.846096234
#> 99 perf 2 mcc 0.0708881205 0.38226007 0.57735027 0.573525244
#> 100 perf 2 fscore 0.0000000000 0.66666667 0.76335878 0.710427365
#> 101 random 3 rank 0.0000000000 0.25000000 0.50000000 0.500000000
#> 102 random 3 score -3.6250298604 -0.75541730 -0.02302904 -0.037024024
#> 103 random 3 label -1.0000000000 -1.00000000 0.00000000 0.000000000
#> 104 random 3 err 0.4900000000 0.52000000 0.54000000 0.539800995
#> 105 random 3 acc 0.4150000000 0.44000000 0.46000000 0.460199005
#> 106 random 3 sp 0.0000000000 0.23000000 0.43000000 0.460199005
#> 107 random 3 sn 0.0000000000 0.17000000 0.43000000 0.460199005
#> 108 random 3 prec 0.0000000000 0.41573034 0.44859813 0.437448276
#> 109 random 3 mcc -0.1950162042 -0.12713096 -0.10356163 -0.096609636
#> 110 random 3 fscore 0.0000000000 0.22666667 0.43434343 0.407378029
#> 111 poor_er 3 rank 0.0000000000 0.25000000 0.50000000 0.500000000
#> 112 poor_er 3 score 0.0004293821 0.44734714 0.67670557 0.636193239
#> 113 poor_er 3 label -1.0000000000 -1.00000000 0.00000000 0.000000000
#> 114 poor_er 3 err 0.2250000000 0.25500000 0.30500000 0.328557214
#> 115 poor_er 3 acc 0.4950000000 0.60500000 0.69500000 0.671442786
#> 116 poor_er 3 sp 0.0000000000 0.47000000 0.76000000 0.671442786
#> 117 poor_er 3 sn 0.0000000000 0.43000000 0.76000000 0.671442786
#> 118 poor_er 3 prec 0.0000000000 0.63398693 0.75000000 0.728944626
#> 119 poor_er 3 mcc -0.0708881205 0.32449720 0.45412814 0.404233319
#> 120 poor_er 3 fscore 0.0000000000 0.57333333 0.71508380 0.627916339
#> 121 good_er 3 rank 0.0000000000 0.25000000 0.50000000 0.500000000
#> 122 good_er 3 score 0.0010490299 0.09118965 0.29093705 0.355893963
#> 123 good_er 3 label -1.0000000000 -1.00000000 0.00000000 0.000000000
#> 124 good_er 3 err 0.2250000000 0.24500000 0.30000000 0.328606965
#> 125 good_er 3 acc 0.4950000000 0.60000000 0.70000000 0.671393035
#> 126 good_er 3 sp 0.0000000000 0.43000000 0.76000000 0.671393035
#> 127 good_er 3 sn 0.0000000000 0.48000000 0.76000000 0.671393035
#> 128 good_er 3 prec 0.4974874372 0.62000000 0.76470588 0.770535440
#> 129 good_er 3 mcc -0.0708881205 0.29552426 0.45068529 0.399076240
#> 130 good_er 3 fscore 0.0000000000 0.64000000 0.71161049 0.635994463
#> 131 excel 3 rank 0.0000000000 0.25000000 0.50000000 0.500000000
#> 132 excel 3 score -2.3114252883 0.01161712 1.50754409 1.541442540
#> 133 excel 3 label -1.0000000000 -1.00000000 0.00000000 0.000000000
#> 134 excel 3 err 0.0400000000 0.12500000 0.25000000 0.254875622
#> 135 excel 3 acc 0.5000000000 0.62500000 0.75000000 0.745124378
#> 136 excel 3 sp 0.0000000000 0.50000000 0.94000000 0.745124378
#> 137 excel 3 sn 0.0000000000 0.50000000 0.94000000 0.745124378
#> 138 excel 3 prec 0.5000000000 0.66666667 0.94174757 0.842598462
#> 139 excel 3 mcc 0.0708881205 0.38226007 0.57735027 0.566164122
#> 140 excel 3 fscore 0.0000000000 0.66666667 0.76335878 0.706867538
#> 141 perf 3 rank 0.0000000000 0.25000000 0.50000000 0.500000000
#> 142 perf 3 score 0.0000000000 0.00000000 0.50000000 0.500000000
#> 143 perf 3 label -1.0000000000 -1.00000000 0.00000000 0.000000000
#> 144 perf 3 err 0.0000000000 0.12500000 0.25000000 0.251243781
#> 145 perf 3 acc 0.5000000000 0.62500000 0.75000000 0.748756219
#> 146 perf 3 sp 0.0000000000 0.50000000 1.00000000 0.748756219
#> 147 perf 3 sn 0.0000000000 0.50000000 1.00000000 0.748756219
#> 148 perf 3 prec 0.5000000000 0.66666667 1.00000000 0.846096234
#> 149 perf 3 mcc 0.0708881205 0.38226007 0.57735027 0.573525244
#> 150 perf 3 fscore 0.0000000000 0.66666667 0.76335878 0.710427365
#> 151 random 4 rank 0.0000000000 0.25000000 0.50000000 0.500000000
#> 152 random 4 score -2.0109648913 -0.70134881 -0.15973021 -0.007720345
#> 153 random 4 label -1.0000000000 -1.00000000 0.00000000 0.000000000
#> 154 random 4 err 0.4500000000 0.48000000 0.49500000 0.496218905
#> 155 random 4 acc 0.4500000000 0.48500000 0.50500000 0.503781095
#> 156 random 4 sp 0.0000000000 0.26000000 0.53000000 0.503781095
#> 157 random 4 sn 0.0000000000 0.23000000 0.53000000 0.503781095
#> 158 random 4 prec 0.0000000000 0.47368421 0.50505051 0.481331137
#> 159 random 4 mcc -0.1331087170 -0.04129094 0.02001602 0.005862552
#> 160 random 4 fscore 0.0000000000 0.30666667 0.52791878 0.450645052
#> 161 poor_er 4 rank 0.0000000000 0.25000000 0.50000000 0.500000000
#> 162 poor_er 4 score 0.0043902281 0.48011132 0.72609675 0.660563583
#> 163 poor_er 4 label -1.0000000000 -1.00000000 0.00000000 0.000000000
#> 164 poor_er 4 err 0.2450000000 0.28000000 0.32500000 0.350199005
#> 165 poor_er 4 acc 0.5000000000 0.58500000 0.67500000 0.649800995
#> 166 poor_er 4 sp 0.0000000000 0.46000000 0.72000000 0.649800995
#> 167 poor_er 4 sn 0.0000000000 0.37000000 0.72000000 0.649800995
#> 168 poor_er 4 prec 0.5000000000 0.63636364 0.72072072 0.693284489
#> 169 poor_er 4 mcc 0.0411345035 0.26889940 0.38917195 0.353634721
#> 170 poor_er 4 fscore 0.0000000000 0.49333333 0.69518717 0.600970690
#> 171 good_er 4 rank 0.0000000000 0.25000000 0.50000000 0.500000000
#> 172 good_er 4 score 0.0013394807 0.09413605 0.25222014 0.313100524
#> 173 good_er 4 label -1.0000000000 -1.00000000 0.00000000 0.000000000
#> 174 good_er 4 err 0.2400000000 0.27500000 0.32000000 0.348258706
#> 175 good_er 4 acc 0.5000000000 0.57500000 0.68000000 0.651741294
#> 176 good_er 4 sp 0.0000000000 0.39000000 0.74000000 0.651741294
#> 177 good_er 4 sn 0.0000000000 0.47000000 0.74000000 0.651741294
#> 178 good_er 4 prec 0.5000000000 0.59602649 0.73958333 0.746618841
#> 179 good_er 4 mcc 0.0320256308 0.25534571 0.39932615 0.354107970
#> 180 good_er 4 fscore 0.0000000000 0.62251656 0.68939394 0.616335707
#> 181 excel 4 rank 0.0000000000 0.25000000 0.50000000 0.500000000
#> 182 excel 4 score -2.1951891607 -0.09192387 1.60889400 1.540674290
#> 183 excel 4 label -1.0000000000 -1.00000000 0.00000000 0.000000000
#> 184 excel 4 err 0.0650000000 0.13000000 0.25000000 0.258557214
#> 185 excel 4 acc 0.5000000000 0.62500000 0.75000000 0.741442786
#> 186 excel 4 sp 0.0000000000 0.50000000 0.93000000 0.741442786
#> 187 excel 4 sn 0.0000000000 0.50000000 0.93000000 0.741442786
#> 188 excel 4 prec 0.5000000000 0.66666667 0.92929293 0.838948129
#> 189 excel 4 mcc 0.0708881205 0.38226007 0.57735027 0.558672302
#> 190 excel 4 fscore 0.0000000000 0.66666667 0.76335878 0.703213223
#> 191 perf 4 rank 0.0000000000 0.25000000 0.50000000 0.500000000
#> 192 perf 4 score 0.0000000000 0.00000000 0.50000000 0.500000000
#> 193 perf 4 label -1.0000000000 -1.00000000 0.00000000 0.000000000
#> 194 perf 4 err 0.0000000000 0.12500000 0.25000000 0.251243781
#> 195 perf 4 acc 0.5000000000 0.62500000 0.75000000 0.748756219
#> 196 perf 4 sp 0.0000000000 0.50000000 1.00000000 0.748756219
#> 197 perf 4 sn 0.0000000000 0.50000000 1.00000000 0.748756219
#> 198 perf 4 prec 0.5000000000 0.66666667 1.00000000 0.846096234
#> 199 perf 4 mcc 0.0708881205 0.38226007 0.57735027 0.573525244
#> 200 perf 4 fscore 0.0000000000 0.66666667 0.76335878 0.710427365
#> 3rd Qu. Max.
#> 1 0.75000000 1.00000000
#> 2 0.59672668 2.61933533
#> 3 1.00000000 1.00000000
#> 4 0.55000000 0.58000000
#> 5 0.50000000 0.54000000
#> 6 0.69000000 1.00000000
#> 7 0.78000000 1.00000000
#> 8 0.50000000 0.52597403
#> 9 0.00000000 0.09504969
#> 10 0.62151394 0.66889632
#> 11 0.75000000 1.00000000
#> 12 0.88751128 0.99966058
#> 13 1.00000000 1.00000000
#> 14 0.39500000 0.50000000
#> 15 0.73500000 0.78000000
#> 16 0.91000000 1.00000000
#> 17 0.95000000 1.00000000
#> 18 0.82978723 1.00000000
#> 19 0.49419373 0.56044854
#> 20 0.76190476 0.78431373
#> 21 0.75000000 1.00000000
#> 22 0.55612298 0.99874166
#> 23 1.00000000 1.00000000
#> 24 0.40000000 0.50000000
#> 25 0.70500000 0.76000000
#> 26 0.97000000 1.00000000
#> 27 0.90000000 1.00000000
#> 28 0.93750000 1.00000000
#> 29 0.46283280 0.53306004
#> 30 0.71204188 0.73214286
#> 31 0.75000000 1.00000000
#> 32 2.99150770 5.55739973
#> 33 1.00000000 1.00000000
#> 34 0.37500000 0.50000000
#> 35 0.87000000 0.93500000
#> 36 1.00000000 1.00000000
#> 37 0.99000000 1.00000000
#> 38 1.00000000 1.00000000
#> 39 0.76043233 0.87354505
#> 40 0.86516854 0.93264249
#> 41 0.75000000 1.00000000
#> 42 1.00000000 1.00000000
#> 43 1.00000000 1.00000000
#> 44 0.37500000 0.50000000
#> 45 0.87500000 1.00000000
#> 46 1.00000000 1.00000000
#> 47 1.00000000 1.00000000
#> 48 1.00000000 1.00000000
#> 49 0.77459667 1.00000000
#> 50 0.87640449 1.00000000
#> 51 0.75000000 1.00000000
#> 52 0.54899881 2.42367377
#> 53 1.00000000 1.00000000
#> 54 0.50000000 0.53500000
#> 55 0.53000000 0.56500000
#> 56 0.80000000 1.00000000
#> 57 0.75000000 1.00000000
#> 58 0.57575758 1.00000000
#> 59 0.09171807 0.17586311
#> 60 0.59848485 0.66889632
#> 61 0.75000000 1.00000000
#> 62 0.88528788 0.99808724
#> 63 1.00000000 1.00000000
#> 64 0.40500000 0.50500000
#> 65 0.69500000 0.73500000
#> 66 0.92000000 1.00000000
#> 67 0.91000000 1.00000000
#> 68 0.78787879 0.84905660
#> 69 0.40856605 0.48071938
#> 70 0.71875000 0.74074074
#> 71 0.75000000 1.00000000
#> 72 0.52515682 0.98672660
#> 73 1.00000000 1.00000000
#> 74 0.40000000 0.50000000
#> 75 0.75500000 0.80000000
#> 76 0.95000000 1.00000000
#> 77 0.93000000 1.00000000
#> 78 0.90909091 1.00000000
#> 79 0.52388619 0.60436722
#> 80 0.76595745 0.79591837
#> 81 0.75000000 1.00000000
#> 82 3.02242254 5.20445617
#> 83 1.00000000 1.00000000
#> 84 0.37500000 0.50000000
#> 85 0.87500000 0.96500000
#> 86 1.00000000 1.00000000
#> 87 1.00000000 1.00000000
#> 88 1.00000000 1.00000000
#> 89 0.77459667 0.93041878
#> 90 0.87640449 0.96551724
#> 91 0.75000000 1.00000000
#> 92 1.00000000 1.00000000
#> 93 1.00000000 1.00000000
#> 94 0.37500000 0.50000000
#> 95 0.87500000 1.00000000
#> 96 1.00000000 1.00000000
#> 97 1.00000000 1.00000000
#> 98 1.00000000 1.00000000
#> 99 0.77459667 1.00000000
#> 100 0.87640449 1.00000000
#> 101 0.75000000 1.00000000
#> 102 0.58458647 2.68975121
#> 103 1.00000000 1.00000000
#> 104 0.56000000 0.58500000
#> 105 0.48000000 0.51000000
#> 106 0.67000000 1.00000000
#> 107 0.73000000 1.00000000
#> 108 0.48275862 0.66666667
#> 109 -0.06907896 0.05862104
#> 110 0.58400000 0.66666667
#> 111 0.75000000 1.00000000
#> 112 0.90289220 0.99902282
#> 113 1.00000000 1.00000000
#> 114 0.39500000 0.50500000
#> 115 0.74500000 0.77500000
#> 116 0.93000000 1.00000000
#> 117 0.97000000 1.00000000
#> 118 0.84444444 0.94117647
#> 119 0.50860135 0.55068879
#> 120 0.76470588 0.78838174
#> 121 0.75000000 1.00000000
#> 122 0.55851272 0.98668399
#> 123 1.00000000 1.00000000
#> 124 0.40000000 0.50500000
#> 125 0.75500000 0.77500000
#> 126 0.98000000 1.00000000
#> 127 0.93000000 1.00000000
#> 128 0.95918367 1.00000000
#> 129 0.52597937 0.58713656
#> 130 0.75132275 0.77669903
#> 131 0.75000000 1.00000000
#> 132 3.00407371 5.98750475
#> 133 1.00000000 1.00000000
#> 134 0.37500000 0.50000000
#> 135 0.87500000 0.96000000
#> 136 1.00000000 1.00000000
#> 137 1.00000000 1.00000000
#> 138 1.00000000 1.00000000
#> 139 0.77459667 0.92295821
#> 140 0.87640449 0.95833333
#> 141 0.75000000 1.00000000
#> 142 1.00000000 1.00000000
#> 143 1.00000000 1.00000000
#> 144 0.37500000 0.50000000
#> 145 0.87500000 1.00000000
#> 146 1.00000000 1.00000000
#> 147 1.00000000 1.00000000
#> 148 1.00000000 1.00000000
#> 149 0.77459667 1.00000000
#> 150 0.87640449 1.00000000
#> 151 0.75000000 1.00000000
#> 152 0.64374019 3.41573715
#> 153 1.00000000 1.00000000
#> 154 0.51500000 0.55000000
#> 155 0.52000000 0.55000000
#> 156 0.73000000 1.00000000
#> 157 0.76000000 1.00000000
#> 158 0.51748252 0.57142857
#> 159 0.05431945 0.10140924
#> 160 0.60408163 0.67118644
#> 161 0.75000000 1.00000000
#> 162 0.90250785 0.99885613
#> 163 1.00000000 1.00000000
#> 164 0.41500000 0.50000000
#> 165 0.72000000 0.75500000
#> 166 0.87000000 1.00000000
#> 167 0.96000000 1.00000000
#> 168 0.75471698 1.00000000
#> 169 0.46136222 0.51530255
#> 170 0.75000000 0.78008299
#> 171 0.75000000 1.00000000
#> 172 0.45467662 0.97708698
#> 173 1.00000000 1.00000000
#> 174 0.42500000 0.50000000
#> 175 0.72500000 0.76000000
#> 176 0.97000000 1.00000000
#> 177 0.89000000 1.00000000
#> 178 0.91304348 1.00000000
#> 179 0.48120488 0.54510812
#> 180 0.72432432 0.75490196
#> 181 0.75000000 1.00000000
#> 182 3.10702141 4.93076999
#> 183 1.00000000 1.00000000
#> 184 0.37500000 0.50000000
#> 185 0.87000000 0.93500000
#> 186 1.00000000 1.00000000
#> 187 1.00000000 1.00000000
#> 188 1.00000000 1.00000000
#> 189 0.76431763 0.87354505
#> 190 0.87336245 0.93193717
#> 191 0.75000000 1.00000000
#> 192 1.00000000 1.00000000
#> 193 1.00000000 1.00000000
#> 194 0.37500000 0.50000000
#> 195 0.87500000 1.00000000
#> 196 1.00000000 1.00000000
#> 197 1.00000000 1.00000000
#> 198 1.00000000 1.00000000
#> 199 0.77459667 1.00000000
#> 200 0.87640449 1.00000000
#>
#>
#> === Input data ===
#>
#> Model name Dataset ID # of negatives # of positives
#> 1 random 1 100 100
#> 2 poor_er 1 100 100
#> 3 good_er 1 100 100
#> 4 excel 1 100 100
#> 5 perf 1 100 100
#> 6 random 2 100 100
#> 7 poor_er 2 100 100
#> 8 good_er 2 100 100
#> 9 excel 2 100 100
#> 10 perf 2 100 100
#> 11 random 3 100 100
#> 12 poor_er 3 100 100
#> 13 good_er 3 100 100
#> 14 excel 3 100 100
#> 15 perf 3 100 100
#> 16 random 4 100 100
#> 17 poor_er 4 100 100
#> 18 good_er 4 100 100
#> 19 excel 4 100 100
#> 20 perf 4 100 100
#>
##################################################
### N-fold cross validation datasets
###
## Load test data
data(M2N50F5)
## Speficy nessesary columns to create mdat
cvdat <- mmdata(
nfold_df = M2N50F5, score_cols = c(1, 2),
lab_col = 3, fold_col = 4,
modnames = c("m1", "m2"), dsids = 1:5
)
## Generate an mmcurve object that contains ROC and Precision-Recall curves
cvcurves <- evalmod(cvdat)
cvcurves
#>
#> === AUCs ===
#>
#> Model name Dataset ID Curve type AUC
#> 1 m1 1 ROC 1.0000000
#> 2 m1 1 PRC 1.0000000
#> 3 m1 2 ROC 0.4166667
#> 4 m1 2 PRC 0.5164199
#> 5 m1 3 ROC 0.2000000
#> 6 m1 3 PRC 0.4891743
#> 7 m1 4 ROC 0.7916667
#> 8 m1 4 PRC 0.7728152
#> 9 m1 5 ROC 0.4400000
#> 10 m1 5 PRC 0.4266312
#> 11 m2 1 ROC 0.4000000
#> 12 m2 1 PRC 0.4247188
#> 13 m2 2 ROC 0.7083333
#> 14 m2 2 PRC 0.6568625
#> 15 m2 3 ROC 0.8400000
#> 16 m2 3 PRC 0.9057736
#> 17 m2 4 ROC 0.7916667
#> 18 m2 4 PRC 0.8527712
#> 19 m2 5 ROC 0.4000000
#> 20 m2 5 PRC 0.4247188
#>
#>
#> === Input data ===
#>
#> Model name Dataset ID # of negatives # of positives
#> 1 m1 1 5 5
#> 2 m1 2 4 6
#> 3 m1 3 5 5
#> 4 m1 4 6 4
#> 5 m1 5 5 5
#> 6 m2 1 5 5
#> 7 m2 2 4 6
#> 8 m2 3 5 5
#> 9 m2 4 6 4
#> 10 m2 5 5 5
#>
## Generate an mmpoints object that contains basic evaluation measures
cvpoints <- evalmod(cvdat, mode = "basic")
cvpoints
#>
#> === Basic performance evaluation measures ===
#>
#> ## Performance measures (Meas.)
#> rank: normalized rank
#> score: score
#> label: label
#> err: error rate
#> acc: accuracy
#> sp: specificity
#> sn: sensitivity
#> prec: precision
#> mcc: Matthews correlation coefficient
#> fscore: F-score
#>
#>
#> Model ID Meas. Min. 1st Qu. Median Mean
#> 1 m1 1 rank 0.0000000 0.25000000 0.500000000 0.500000000
#> 2 m1 1 score -1.5761733 -0.92376396 -0.002327284 0.113500523
#> 3 m1 1 label -1.0000000 -1.00000000 0.000000000 0.000000000
#> 4 m1 1 err 0.0000000 0.15000000 0.300000000 0.272727273
#> 5 m1 1 acc 0.5000000 0.60000000 0.700000000 0.727272727
#> 6 m1 1 sp 0.0000000 0.50000000 1.000000000 0.727272727
#> 7 m1 1 sn 0.0000000 0.50000000 1.000000000 0.727272727
#> 8 m1 1 prec 0.5000000 0.66964286 1.000000000 0.838924964
#> 9 m1 1 mcc 0.3333333 0.50000000 0.654653671 0.623218574
#> 10 m1 1 fscore 0.0000000 0.61904762 0.750000000 0.676023471
#> 11 m1 2 rank 0.0000000 0.25000000 0.500000000 0.500000000
#> 12 m1 2 score -1.8425134 -1.08765052 -0.450813882 -0.375652956
#> 13 m1 2 label -1.0000000 -1.00000000 1.000000000 0.200000000
#> 14 m1 2 err 0.4000000 0.45000000 0.500000000 0.536363636
#> 15 m1 2 acc 0.3000000 0.40000000 0.500000000 0.463636364
#> 16 m1 2 sp 0.0000000 0.25000000 0.500000000 0.454545455
#> 17 m1 2 sn 0.0000000 0.16666667 0.500000000 0.469696970
#> 18 m1 2 prec 0.0000000 0.41666667 0.555555556 0.450180375
#> 19 m1 2 mcc -0.4082483 -0.27216553 -0.102062073 -0.125094358
#> 20 m1 2 fscore 0.0000000 0.23611111 0.545454545 0.439152766
#> 21 m1 3 rank 0.0000000 0.25000000 0.500000000 0.500000000
#> 22 m1 3 score -1.1558752 -0.16251342 0.389025335 0.414373732
#> 23 m1 3 label -1.0000000 -1.00000000 0.000000000 0.000000000
#> 24 m1 3 err 0.4000000 0.50000000 0.600000000 0.636363636
#> 25 m1 3 acc 0.1000000 0.25000000 0.400000000 0.363636364
#> 26 m1 3 sp 0.0000000 0.00000000 0.200000000 0.363636364
#> 27 m1 3 sn 0.0000000 0.20000000 0.200000000 0.363636364
#> 28 m1 3 prec 0.1666667 0.26785714 0.375000000 0.459559885
#> 29 m1 3 mcc -0.8164966 -0.60000000 -0.408248290 -0.355290715
#> 30 m1 3 fscore 0.0000000 0.21111111 0.285714286 0.318732278
#> 31 m1 4 rank 0.0000000 0.25000000 0.500000000 0.500000000
#> 32 m1 4 score -1.5462252 -1.19762686 -0.400691531 -0.141998509
#> 33 m1 4 label -1.0000000 -1.00000000 -1.000000000 -0.200000000
#> 34 m1 4 err 0.2000000 0.30000000 0.400000000 0.372727273
#> 35 m1 4 acc 0.4000000 0.60000000 0.600000000 0.627272727
#> 36 m1 4 sp 0.0000000 0.41666667 0.666666667 0.606060606
#> 37 m1 4 sn 0.0000000 0.50000000 0.750000000 0.659090909
#> 38 m1 4 prec 0.4000000 0.50000000 0.571428571 0.652958153
#> 39 m1 4 mcc 0.1666667 0.27216553 0.408248290 0.379646701
#> 40 m1 4 fscore 0.0000000 0.53571429 0.600000000 0.544137681
#> 41 m1 5 rank 0.0000000 0.25000000 0.500000000 0.500000000
#> 42 m1 5 score -1.5944199 -0.50932491 -0.113562820 0.006988569
#> 43 m1 5 label -1.0000000 -1.00000000 0.000000000 0.000000000
#> 44 m1 5 err 0.4000000 0.50000000 0.500000000 0.527272727
#> 45 m1 5 acc 0.3000000 0.40000000 0.500000000 0.472727273
#> 46 m1 5 sp 0.0000000 0.30000000 0.400000000 0.472727273
#> 47 m1 5 sn 0.0000000 0.10000000 0.400000000 0.472727273
#> 48 m1 5 prec 0.0000000 0.16666667 0.500000000 0.350937951
#> 49 m1 5 mcc -0.5000000 -0.21821789 0.000000000 -0.077777778
#> 50 m1 5 fscore 0.0000000 0.12500000 0.444444444 0.391172968
#> 51 m2 1 rank 0.0000000 0.25000000 0.500000000 0.500000000
#> 52 m2 1 score -1.5666375 -0.93714170 -0.338617003 -0.081054576
#> 53 m2 1 label -1.0000000 -1.00000000 0.000000000 0.000000000
#> 54 m2 1 err 0.5000000 0.50000000 0.500000000 0.545454545
#> 55 m2 1 acc 0.4000000 0.40000000 0.500000000 0.454545455
#> 56 m2 1 sp 0.0000000 0.20000000 0.400000000 0.454545455
#> 57 m2 1 sn 0.0000000 0.20000000 0.400000000 0.454545455
#> 58 m2 1 prec 0.0000000 0.36666667 0.444444444 0.373304473
#> 59 m2 1 mcc -0.3333333 -0.21821789 -0.200000000 -0.144789161
#> 60 m2 1 fscore 0.0000000 0.26785714 0.444444444 0.389008466
#> 61 m2 2 rank 0.0000000 0.25000000 0.500000000 0.500000000
#> 62 m2 2 score -0.9422955 -0.51338997 0.168580899 -0.003990330
#> 63 m2 2 label -1.0000000 -1.00000000 1.000000000 0.200000000
#> 64 m2 2 err 0.2000000 0.30000000 0.400000000 0.409090909
#> 65 m2 2 acc 0.3000000 0.45000000 0.600000000 0.590909091
#> 66 m2 2 sp 0.0000000 0.50000000 0.750000000 0.613636364
#> 67 m2 2 sn 0.0000000 0.25000000 0.666666667 0.575757576
#> 68 m2 2 prec 0.0000000 0.55000000 0.666666667 0.570995671
#> 69 m2 2 mcc -0.4082483 0.08908708 0.356348323 0.244147488
#> 70 m2 2 fscore 0.0000000 0.34722222 0.727272727 0.548311285
#> 71 m2 3 rank 0.0000000 0.25000000 0.500000000 0.500000000
#> 72 m2 3 score -1.0962763 -0.40057827 -0.167608702 0.082174350
#> 73 m2 3 label -1.0000000 -1.00000000 0.000000000 0.000000000
#> 74 m2 3 err 0.1000000 0.25000000 0.400000000 0.345454545
#> 75 m2 3 acc 0.5000000 0.55000000 0.600000000 0.654545455
#> 76 m2 3 sp 0.0000000 0.30000000 0.800000000 0.654545455
#> 77 m2 3 sn 0.0000000 0.50000000 0.800000000 0.654545455
#> 78 m2 3 prec 0.5000000 0.56349206 0.800000000 0.781240981
#> 79 m2 3 mcc 0.0000000 0.33333333 0.408248290 0.429364789
#> 80 m2 3 fscore 0.0000000 0.59340659 0.666666667 0.612175199
#> 81 m2 4 rank 0.0000000 0.25000000 0.500000000 0.500000000
#> 82 m2 4 score -1.1654721 -0.35410360 -0.099106516 0.103674375
#> 83 m2 4 label -1.0000000 -1.00000000 -1.000000000 -0.200000000
#> 84 m2 4 err 0.1000000 0.25000000 0.400000000 0.372727273
#> 85 m2 4 acc 0.4000000 0.50000000 0.600000000 0.627272727
#> 86 m2 4 sp 0.0000000 0.25000000 0.666666667 0.606060606
#> 87 m2 4 sn 0.0000000 0.62500000 0.750000000 0.659090909
#> 88 m2 4 prec 0.3750000 0.43650794 0.600000000 0.681637807
#> 89 m2 4 mcc -0.1020621 0.25000000 0.408248290 0.369241846
#> 90 m2 4 fscore 0.0000000 0.52272727 0.600000000 0.561158538
#> 91 m2 5 rank 0.0000000 0.25000000 0.500000000 0.500000000
#> 92 m2 5 score -1.7908147 -1.14209817 0.032207934 -0.329061088
#> 93 m2 5 label -1.0000000 -1.00000000 0.000000000 0.000000000
#> 94 m2 5 err 0.5000000 0.50000000 0.500000000 0.545454545
#> 95 m2 5 acc 0.4000000 0.40000000 0.500000000 0.454545455
#> 96 m2 5 sp 0.0000000 0.20000000 0.400000000 0.454545455
#> 97 m2 5 sn 0.0000000 0.20000000 0.400000000 0.454545455
#> 98 m2 5 prec 0.0000000 0.36666667 0.444444444 0.373304473
#> 99 m2 5 mcc -0.3333333 -0.21821789 -0.200000000 -0.144789161
#> 100 m2 5 fscore 0.0000000 0.26785714 0.444444444 0.389008466
#> 3rd Qu. Max.
#> 1 0.75000000 1.0000000
#> 2 1.06708368 2.0606025
#> 3 1.00000000 1.0000000
#> 4 0.40000000 0.5000000
#> 5 0.85000000 1.0000000
#> 6 1.00000000 1.0000000
#> 7 1.00000000 1.0000000
#> 8 1.00000000 1.0000000
#> 9 0.81649658 1.0000000
#> 10 0.86111111 1.0000000
#> 11 0.75000000 1.0000000
#> 12 -0.08830171 2.2002973
#> 13 1.00000000 1.0000000
#> 14 0.60000000 0.7000000
#> 15 0.55000000 0.6000000
#> 16 0.62500000 1.0000000
#> 17 0.75000000 1.0000000
#> 18 0.60000000 0.6666667
#> 19 0.00000000 0.1666667
#> 20 0.66666667 0.7500000
#> 21 0.75000000 1.0000000
#> 22 1.01753386 1.6870336
#> 23 1.00000000 1.0000000
#> 24 0.75000000 0.9000000
#> 25 0.50000000 0.6000000
#> 26 0.70000000 1.0000000
#> 27 0.50000000 1.0000000
#> 28 0.50000000 1.0000000
#> 29 -0.21821789 0.3333333
#> 30 0.39743590 0.6666667
#> 31 0.75000000 1.0000000
#> 32 1.00867637 1.5387145
#> 33 1.00000000 1.0000000
#> 34 0.40000000 0.6000000
#> 35 0.70000000 0.8000000
#> 36 0.91666667 1.0000000
#> 37 1.00000000 1.0000000
#> 38 0.83333333 1.0000000
#> 39 0.40824829 0.6123724
#> 40 0.66666667 0.7272727
#> 41 0.75000000 1.0000000
#> 42 0.45270677 1.9237289
#> 43 1.00000000 1.0000000
#> 44 0.60000000 0.7000000
#> 45 0.50000000 0.6000000
#> 46 0.60000000 1.0000000
#> 47 0.80000000 1.0000000
#> 48 0.50000000 0.5714286
#> 49 0.00000000 0.3333333
#> 50 0.64102564 0.7142857
#> 51 0.75000000 1.0000000
#> 52 0.43266892 2.8510651
#> 53 1.00000000 1.0000000
#> 54 0.60000000 0.6000000
#> 55 0.50000000 0.5000000
#> 56 0.70000000 1.0000000
#> 57 0.70000000 1.0000000
#> 58 0.50000000 0.5000000
#> 59 0.00000000 0.0000000
#> 60 0.55844156 0.6666667
#> 61 0.75000000 1.0000000
#> 62 0.39797893 0.7890534
#> 63 1.00000000 1.0000000
#> 64 0.55000000 0.7000000
#> 65 0.70000000 0.8000000
#> 66 0.75000000 1.0000000
#> 67 0.91666667 1.0000000
#> 68 0.75000000 0.8333333
#> 69 0.40824829 0.6123724
#> 70 0.78461538 0.8571429
#> 71 0.75000000 1.0000000
#> 72 0.35464019 2.0647833
#> 73 1.00000000 1.0000000
#> 74 0.45000000 0.5000000
#> 75 0.75000000 0.9000000
#> 76 1.00000000 1.0000000
#> 77 0.80000000 1.0000000
#> 78 1.00000000 1.0000000
#> 79 0.60000000 0.8164966
#> 80 0.73863636 0.8888889
#> 81 0.75000000 1.0000000
#> 82 0.79072036 1.4431452
#> 83 1.00000000 1.0000000
#> 84 0.50000000 0.6000000
#> 85 0.75000000 0.9000000
#> 86 1.00000000 1.0000000
#> 87 0.75000000 1.0000000
#> 88 1.00000000 1.0000000
#> 89 0.58333333 0.8017837
#> 90 0.66666667 0.8571429
#> 91 0.75000000 1.0000000
#> 92 0.31323748 0.7002948
#> 93 1.00000000 1.0000000
#> 94 0.60000000 0.6000000
#> 95 0.50000000 0.5000000
#> 96 0.70000000 1.0000000
#> 97 0.70000000 1.0000000
#> 98 0.50000000 0.5000000
#> 99 0.00000000 0.0000000
#> 100 0.55844156 0.6666667
#>
#>
#> === Input data ===
#>
#> Model name Dataset ID # of negatives # of positives
#> 1 m1 1 5 5
#> 2 m1 2 4 6
#> 3 m1 3 5 5
#> 4 m1 4 6 4
#> 5 m1 5 5 5
#> 6 m2 1 5 5
#> 7 m2 2 4 6
#> 8 m2 3 5 5
#> 9 m2 4 6 4
#> 10 m2 5 5 5
#>
## Specify mmdata arguments from evalmod
cvcurves2 <- evalmod(
nfold_df = M2N50F5, score_cols = c(1, 2),
lab_col = 3, fold_col = 4,
modnames = c("m1", "m2"), dsids = 1:5
)
cvcurves2
#>
#> === AUCs ===
#>
#> Model name Dataset ID Curve type AUC
#> 1 m1 1 ROC 1.0000000
#> 2 m1 1 PRC 1.0000000
#> 3 m1 2 ROC 0.4166667
#> 4 m1 2 PRC 0.5164199
#> 5 m1 3 ROC 0.2000000
#> 6 m1 3 PRC 0.4891743
#> 7 m1 4 ROC 0.7916667
#> 8 m1 4 PRC 0.7728152
#> 9 m1 5 ROC 0.4400000
#> 10 m1 5 PRC 0.4266312
#> 11 m2 1 ROC 0.4000000
#> 12 m2 1 PRC 0.4247188
#> 13 m2 2 ROC 0.7083333
#> 14 m2 2 PRC 0.6568625
#> 15 m2 3 ROC 0.8400000
#> 16 m2 3 PRC 0.9057736
#> 17 m2 4 ROC 0.7916667
#> 18 m2 4 PRC 0.8527712
#> 19 m2 5 ROC 0.4000000
#> 20 m2 5 PRC 0.4247188
#>
#>
#> === Input data ===
#>
#> Model name Dataset ID # of negatives # of positives
#> 1 m1 1 5 5
#> 2 m1 2 4 6
#> 3 m1 3 5 5
#> 4 m1 4 6 4
#> 5 m1 5 5 5
#> 6 m2 1 5 5
#> 7 m2 2 4 6
#> 8 m2 3 5 5
#> 9 m2 4 6 4
#> 10 m2 5 5 5
#>
##################################################
### AUC with the U statistic
###
## mode = "aucroc" returns 'aucroc' S3 object
data(P10N10)
# 'aucroc' S3 object
uauc1 <- evalmod(
scores = P10N10$scores, labels = P10N10$labels,
mode = "aucroc"
)
# print 'aucroc'
uauc1
#>
#> === Input data ===
#>
#> Model name Dataset ID # of negatives # of positives
#> 1 m1 1 10 10
#>
#>
#> === AUCs ===
#>
#> Model name Dataset ID AUC U
#> 1 m1 1 0.72 72
#>
# as.data.frame 'aucroc'
as.data.frame(uauc1)
#> modnames dsids aucs ustats
#> 1 m1 1 0.72 72
## It is 2-3 times faster than mode = "rocprc"
# A sample of 100,000
samp1 <- create_sim_samples(1, 50000, 50000)
# a function to test mode = "rocprc"
func_evalmod_rocprc <- function(samp) {
curves <- evalmod(scores = samp$scores, labels = samp$labels)
aucs <- auc(curves)
}
# a function to test mode = "aucroc"
func_evalmod_aucroc <- function(samp) {
uaucs <- evalmod(
scores = samp$scores, labels = samp$labels,
mode = "aucroc"
)
as.data.frame(uaucs)
}
# Process time
system.time(res1 <- func_evalmod_rocprc(samp1))
#> user system elapsed
#> 0.029 0.008 0.036
system.time(res2 <- func_evalmod_aucroc(samp1))
#> user system elapsed
#> 0.017 0.000 0.017
# AUCs
res1
#> modnames dsids curvetypes aucs
#> 1 m1 1 ROC 0.4989053
#> 2 m1 1 PRC 0.4985752
res2
#> modnames dsids aucs ustats
#> 1 m1 1 0.4989053 1247263329