The evalmod function calculates ROC and Precision-Recall curves for specified prediction scores and binary labels. It also calculate several basic performance evaluation measures, such as accuracy, error rate, and precision, by specifying mode as "basic".

evalmod(
  mdat,
  mode = NULL,
  scores = NULL,
  labels = NULL,
  modnames = NULL,
  dsids = NULL,
  posclass = NULL,
  na_worst = TRUE,
  ties_method = "equiv",
  calc_avg = TRUE,
  cb_alpha = 0.05,
  raw_curves = FALSE,
  x_bins = 1000,
  ...
)

Arguments

mdat

An S3 object created by the mmdata function. It contains formatted scores and labels. The evalmod function ignores the following arguments when mdat is specified.

  • scores

  • labels

  • modnames

  • dsids

  • posclass

  • na_worst

  • ties_method

These arguments are internally passed to the mmdata function when mdat is unspecified. In that case, both scores and labels must be at least specified.

mode

A string that specifies the types of evaluation measures that the evalmod function calculates.

"rocprc"

ROC and Precision-Recall curves

"prcroc"

Same as above

"basic"

Normalized ranks vs. accuracy, error rate, specificity, sensitivity, precision, Matthews correlation coefficient, and F-score.

"aucroc"

Fast AUC(ROC) calculation with the U statistic

scores

A numeric dataset of predicted scores. It can be a vector, a matrix, an array, a data frame, or a list. The join_scores function can be useful to make scores with multiple datasets.

labels

A numeric, character, logical, or factor dataset of observed labels. It can be a vector, a matrix, an array, a data frame, or a list. The join_labels function can be useful to make labels with multiple datasets.

modnames

A character vector for the names of the models. The evalmod function automatically generates default names as "m1", "m2", "m3", and so on when it is NULL.

dsids

A numeric vector for test dataset IDs. The evalmod function automatically generates the default ID as 1 when it is NULL.

posclass

A scalar value to specify the label of positives in labels. It must be the same data type as labels. For example, posclass = -1 changes the positive label from 1 to -1 when labels contains 1 and -1. The positive label will be automatically detected when posclass is NULL.

na_worst

A Boolean value for controlling the treatment of NAs in scores.

TRUE

All NAs are treated as the worst scores

FALSE

All NAs are treated as the best scores

ties_method

A string for controlling ties in scores.

"equiv"

Ties are equivalently ranked

"first"

Ties are ranked in an increasing order as appeared

"random"

Ties are ranked in random order

calc_avg

A logical value to specify whether average curves should be calculated. It is effective only when dsids contains multiple dataset IDs. For instance, the function calculates the average for the model "m1" when modnames is c("m1", "m1", "m1") and dsids is c(1, 2, 3). The calculation points are defined by x_bins.

cb_alpha

A numeric value with range [0, 1] to specify the alpha value of the point-wise confidence bounds calculation. It is effective only when calc_avg is set to TRUE. For example, it should be 0.05 for the 95% confidence level. The calculation points are defined by x_bins.

raw_curves

A logical value to specify whether all raw curves should be discarded after the average curves are calculated. It is effective only when calc_avg is set to TRUE.

x_bins

An integer value to specify the number of minimum bins on the x-axis. It is then used to define supporting points For instance, the x-values of the supporting points will be c(0, 0.5, 1) and c(0, 0.25, 0.5, 0.75, 1) when x_bins = 2 and x_bins = 4, respectively. All corresponding y-values of the supporting points are calculated.

...

These additional arguments are passed to mmdata for data preparation.

Value

The evalmod function returns an S3 object that contains performance evaluation measures. The number of models and the number of datasets can be controlled by modnames and dsids. For example, the number of models is "single" and the number of test datasets is "multiple" when modnames = c("m1", "m1", "m1") and dsids = c(1, 2, 3) are specified.

Different S3 objects have different default behaviors of S3 generics, such as plot, autoplot, and fortify.

  1. The evalmod function returns one of the following S3 objects when mode is "prcroc". The objects contain ROC and Precision-Recall curves.

    S3 object# of models# of test datasets
    sscurvessinglesingle
    mscurvesmultiplesingle
    smcurvessinglemultiple
    mmcurvesmultiplemultiple
  2. The evalmod function returns one of the following S3 objects when mode is "basic". They contain five different basic evaluation measures; error rate, accuracy, specificity, sensitivity, and precision.

    S3 object# of models# of test datasets
    sspointssinglesingle
    mspointsmultiplesingle
    smpointssinglemultiple
    mmpointsmultiplemultiple
  3. The evalmod function returns the aucroc S3 object when mode is "aucroc", which can be used with 'print' and 'as.data.frame'.

See also

plot for plotting curves with the general R plot. autoplot and fortify for plotting curves with ggplot2. mmdata for formatting input data. join_scores and join_labels for formatting scores and labels with multiple datasets. format_nfold for creating n-fold cross validation dataset from data frame. create_sim_samples for generating random samples for simulations.

Examples

################################################## ### Single model & single test dataset ### ## Load a dataset with 10 positives and 10 negatives data(P10N10) ## Generate an sscurve object that contains ROC and Precision-Recall curves sscurves <- evalmod(scores = P10N10$scores, labels = P10N10$labels) sscurves
#> #> === AUCs === #> #> Model name Dataset ID Curve type AUC #> 1 m1 1 ROC 0.7200000 #> 2 m1 1 PRC 0.7397716 #> #> #> === Input data === #> #> Model name Dataset ID # of negatives # of positives #> 1 m1 1 10 10 #>
## Generate an sspoints object that contains basic evaluation measures sspoints <- evalmod(mode = "basic", scores = P10N10$scores, labels = P10N10$labels) sspoints
#> #> === Basic performance evaluation measures === #> #> ## Performance measures (Meas.) #> rank: normalized rank #> score: score #> label: label #> err: error rate #> acc: accuracy #> sp: specificity #> sn: sensitivity #> prec: precision #> mcc: Matthews correlation coefficient #> fscore: F-score #> #> #> Model ID Meas. Min. 1st Qu. Median Mean 3rd Qu. #> 1 m1 1 rank 0.0000000 0.2500000 0.5000000 0.5000000 0.7500000 #> 2 m1 1 score 5.0000000 5.7500000 14.0000000 11.7500000 15.2500000 #> 3 m1 1 label -1.0000000 -1.0000000 0.0000000 0.0000000 1.0000000 #> 4 m1 1 err 0.3000000 0.3500000 0.4000000 0.3952381 0.4400000 #> 5 m1 1 acc 0.5000000 0.5600000 0.6000000 0.6047619 0.6500000 #> 6 m1 1 sp 0.0000000 0.4000000 0.6333333 0.6047619 0.9000000 #> 7 m1 1 sn 0.0000000 0.4000000 0.6333333 0.6047619 0.9000000 #> 8 m1 1 prec 0.5000000 0.5750000 0.6333333 0.6892147 0.7619048 #> 9 m1 1 mcc 0.1376494 0.2238168 0.2666667 0.2755698 0.3367701 #> 10 m1 1 fscore 0.0000000 0.5333333 0.6333333 0.5579798 0.6758621 #> Max. #> 1 1.0000000 #> 2 20.0000000 #> 3 1.0000000 #> 4 0.5000000 #> 5 0.7000000 #> 6 1.0000000 #> 7 1.0000000 #> 8 1.0000000 #> 9 0.4364358 #> 10 0.7200000 #> #> #> === Input data === #> #> Model name Dataset ID # of negatives # of positives #> 1 m1 1 10 10 #>
################################################## ### Multiple models & single test dataset ### ## Create sample datasets with 100 positives and 100 negatives samps <- create_sim_samples(1, 100, 100, "all") mdat <- mmdata(samps[["scores"]], samps[["labels"]], modnames = samps[["modnames"]]) ## Generate an mscurve object that contains ROC and Precision-Recall curves mscurves <- evalmod(mdat) mscurves
#> #> === AUCs === #> #> Model name Dataset ID Curve type AUC #> 1 random 1 ROC 0.4914000 #> 2 random 1 PRC 0.4981313 #> 3 poor_er 1 ROC 0.8485000 #> 4 poor_er 1 PRC 0.8195495 #> 5 good_er 1 ROC 0.7655000 #> 6 good_er 1 PRC 0.7967180 #> 7 excel 1 ROC 0.9756000 #> 8 excel 1 PRC 0.9812341 #> 9 perf 1 ROC 1.0000000 #> 10 perf 1 PRC 1.0000000 #> #> #> === Input data === #> #> Model name Dataset ID # of negatives # of positives #> 1 random 1 100 100 #> 2 poor_er 1 100 100 #> 3 good_er 1 100 100 #> 4 excel 1 100 100 #> 5 perf 1 100 100 #>
## Generate an mspoints object that contains basic evaluation measures mspoints <- evalmod(mdat, mode = "basic") mspoints
#> #> === Basic performance evaluation measures === #> #> ## Performance measures (Meas.) #> rank: normalized rank #> score: score #> label: label #> err: error rate #> acc: accuracy #> sp: specificity #> sn: sensitivity #> prec: precision #> mcc: Matthews correlation coefficient #> fscore: F-score #> #> #> Model ID Meas. Min. 1st Qu. Median Mean #> 1 random 1 rank 0.000000000 0.25000000 0.50000000 0.5000000 #> 2 random 1 score -2.868475635 -0.64169419 -0.13714006 -0.1438747 #> 3 random 1 label -1.000000000 -1.00000000 0.00000000 0.0000000 #> 4 random 1 err 0.465000000 0.49000000 0.50500000 0.5042786 #> 5 random 1 acc 0.450000000 0.48500000 0.49500000 0.4957214 #> 6 random 1 sp 0.000000000 0.21000000 0.52000000 0.4957214 #> 7 random 1 sn 0.000000000 0.27000000 0.52000000 0.4957214 #> 8 random 1 prec 0.333333333 0.48251748 0.49740933 0.4964357 #> 9 random 1 mcc -0.118812114 -0.04794157 -0.01044903 -0.0147898 #> 10 random 1 fscore 0.000000000 0.35526316 0.51485149 0.4461990 #> 11 poor_er 1 rank 0.000000000 0.25000000 0.50000000 0.5000000 #> 12 poor_er 1 score 0.004389839 0.50576037 0.72147420 0.6588873 #> 13 poor_er 1 label -1.000000000 -1.00000000 0.00000000 0.0000000 #> 14 poor_er 1 err 0.220000000 0.25000000 0.32000000 0.3266169 #> 15 poor_er 1 acc 0.500000000 0.61000000 0.68000000 0.6733831 #> 16 poor_er 1 sp 0.000000000 0.46000000 0.77000000 0.6733831 #> 17 poor_er 1 sn 0.000000000 0.42000000 0.77000000 0.6733831 #> 18 poor_er 1 prec 0.500000000 0.64000000 0.76237624 0.7441101 #> 19 poor_er 1 mcc 0.070888121 0.33929250 0.43643578 0.4092935 #> 20 poor_er 1 fscore 0.000000000 0.56578947 0.72463768 0.6301384 #> 21 good_er 1 rank 0.000000000 0.25000000 0.50000000 0.5000000 #> 22 good_er 1 score 0.001153760 0.11649792 0.30256209 0.3507321 #> 23 good_er 1 label -1.000000000 -1.00000000 0.00000000 0.0000000 #> 24 good_er 1 err 0.290000000 0.31500000 0.34000000 0.3679104 #> 25 good_er 1 acc 0.495000000 0.58500000 0.66000000 0.6320896 #> 26 good_er 1 sp 0.000000000 0.38000000 0.70000000 0.6320896 #> 27 good_er 1 sn 0.000000000 0.43000000 0.70000000 0.6320896 #> 28 good_er 1 prec 0.497461929 0.58823529 0.69696970 0.7224566 #> 29 good_er 1 mcc -0.070888121 0.26197681 0.35682062 0.3130253 #> 30 good_er 1 fscore 0.000000000 0.56578947 0.67132867 0.5945702 #> 31 excel 1 rank 0.000000000 0.25000000 0.50000000 0.5000000 #> 32 excel 1 score -2.356995112 -0.03876904 1.30891590 1.5173648 #> 33 excel 1 label -1.000000000 -1.00000000 0.00000000 0.0000000 #> 34 excel 1 err 0.055000000 0.14000000 0.25500000 0.2633831 #> 35 excel 1 acc 0.500000000 0.62000000 0.74500000 0.7366169 #> 36 excel 1 sp 0.000000000 0.49000000 0.94000000 0.7366169 #> 37 excel 1 sn 0.000000000 0.50000000 0.94000000 0.7366169 #> 38 excel 1 prec 0.500000000 0.66000000 0.94000000 0.8353507 #> 39 excel 1 mcc 0.070888121 0.36927447 0.56965192 0.5477081 #> 40 excel 1 fscore 0.000000000 0.66666667 0.75776398 0.6991233 #> 41 perf 1 rank 0.000000000 0.25000000 0.50000000 0.5000000 #> 42 perf 1 score 0.000000000 0.00000000 0.50000000 0.5000000 #> 43 perf 1 label -1.000000000 -1.00000000 0.00000000 0.0000000 #> 44 perf 1 err 0.000000000 0.12500000 0.25000000 0.2512438 #> 45 perf 1 acc 0.500000000 0.62500000 0.75000000 0.7487562 #> 46 perf 1 sp 0.000000000 0.50000000 1.00000000 0.7487562 #> 47 perf 1 sn 0.000000000 0.50000000 1.00000000 0.7487562 #> 48 perf 1 prec 0.500000000 0.66666667 1.00000000 0.8460962 #> 49 perf 1 mcc 0.070888121 0.38226007 0.57735027 0.5735252 #> 50 perf 1 fscore 0.000000000 0.66666667 0.76335878 0.7104274 #> 3rd Qu. Max. #> 1 0.75000000 1.00000000 #> 2 0.43370152 2.83235561 #> 3 1.00000000 1.00000000 #> 4 0.51500000 0.55000000 #> 5 0.51000000 0.53500000 #> 6 0.77000000 1.00000000 #> 7 0.71000000 1.00000000 #> 8 0.51376147 1.00000000 #> 9 0.02144871 0.07601962 #> 10 0.56790123 0.66666667 #> 11 0.75000000 1.00000000 #> 12 0.89504416 0.99926767 #> 13 1.00000000 1.00000000 #> 14 0.39000000 0.50000000 #> 15 0.75000000 0.78000000 #> 16 0.92000000 1.00000000 #> 17 0.96000000 1.00000000 #> 18 0.84000000 1.00000000 #> 19 0.51489916 0.57154761 #> 20 0.76530612 0.79090909 #> 21 0.75000000 1.00000000 #> 22 0.52141088 0.99778628 #> 23 1.00000000 1.00000000 #> 24 0.41500000 0.50500000 #> 25 0.68500000 0.71000000 #> 26 0.93000000 1.00000000 #> 27 0.88000000 1.00000000 #> 28 0.85714286 1.00000000 #> 29 0.38935791 0.43632197 #> 30 0.69856459 0.72727273 #> 31 0.75000000 1.00000000 #> 32 3.14531194 5.02310259 #> 33 1.00000000 1.00000000 #> 34 0.38000000 0.50000000 #> 35 0.86000000 0.94500000 #> 36 1.00000000 1.00000000 #> 37 0.99000000 1.00000000 #> 38 1.00000000 1.00000000 #> 39 0.74426518 0.89004450 #> 40 0.85840708 0.94472362 #> 41 0.75000000 1.00000000 #> 42 1.00000000 1.00000000 #> 43 1.00000000 1.00000000 #> 44 0.37500000 0.50000000 #> 45 0.87500000 1.00000000 #> 46 1.00000000 1.00000000 #> 47 1.00000000 1.00000000 #> 48 1.00000000 1.00000000 #> 49 0.77459667 1.00000000 #> 50 0.87640449 1.00000000 #> #> #> === Input data === #> #> Model name Dataset ID # of negatives # of positives #> 1 random 1 100 100 #> 2 poor_er 1 100 100 #> 3 good_er 1 100 100 #> 4 excel 1 100 100 #> 5 perf 1 100 100 #>
################################################## ### Single model & multiple test datasets ### ## Create sample datasets with 100 positives and 100 negatives samps <- create_sim_samples(4, 100, 100, "good_er") mdat <- mmdata(samps[["scores"]], samps[["labels"]], modnames = samps[["modnames"]], dsids = samps[["dsids"]]) ## Generate an smcurve object that contains ROC and Precision-Recall curves smcurves <- evalmod(mdat) smcurves
#> #> === AUCs === #> #> Model name Dataset ID Curve type AUC #> 1 good_er 1 ROC 0.8074000 #> 2 good_er 1 PRC 0.8266119 #> 3 good_er 2 ROC 0.7968000 #> 4 good_er 2 PRC 0.8249916 #> 5 good_er 3 ROC 0.8007000 #> 6 good_er 3 PRC 0.8454228 #> 7 good_er 4 ROC 0.7631000 #> 8 good_er 4 PRC 0.8134424 #> #> #> === Input data === #> #> Model name Dataset ID # of negatives # of positives #> 1 good_er 1 100 100 #> 2 good_er 2 100 100 #> 3 good_er 3 100 100 #> 4 good_er 4 100 100 #>
## Generate an smpoints object that contains basic evaluation measures smpoints <- evalmod(mdat, mode = "basic") smpoints
#> #> === Basic performance evaluation measures === #> #> ## Performance measures (Meas.) #> rank: normalized rank #> score: score #> label: label #> err: error rate #> acc: accuracy #> sp: specificity #> sn: sensitivity #> prec: precision #> mcc: Matthews correlation coefficient #> fscore: F-score #> #> #> Model ID Meas. Min. 1st Qu. Median Mean 3rd Qu. #> 1 good_er 1 rank 0.000000000 0.25000000 0.5000000 0.5000000 0.7500000 #> 2 good_er 1 score 0.002220182 0.11855029 0.2660228 0.3440549 0.5408924 #> 3 good_er 1 label -1.000000000 -1.00000000 0.0000000 0.0000000 1.0000000 #> 4 good_er 1 err 0.240000000 0.27500000 0.3300000 0.3470647 0.4200000 #> 5 good_er 1 acc 0.500000000 0.58000000 0.6700000 0.6529353 0.7250000 #> 6 good_er 1 sp 0.000000000 0.40000000 0.7400000 0.6529353 0.9500000 #> 7 good_er 1 sn 0.000000000 0.45000000 0.7400000 0.6529353 0.9000000 #> 8 good_er 1 prec 0.500000000 0.59731544 0.7400000 0.7432531 0.8936170 #> 9 good_er 1 mcc 0.070888121 0.26197681 0.3904344 0.3602167 0.4685964 #> 10 good_er 1 fscore 0.000000000 0.60000000 0.6886447 0.6156131 0.7322404 #> 11 good_er 2 rank 0.000000000 0.25000000 0.5000000 0.5000000 0.7500000 #> 12 good_er 2 score 0.005721929 0.10189603 0.2711515 0.3468729 0.5285391 #> 13 good_er 2 label -1.000000000 -1.00000000 0.0000000 0.0000000 1.0000000 #> 14 good_er 2 err 0.240000000 0.28500000 0.3500000 0.3523383 0.4100000 #> 15 good_er 2 acc 0.500000000 0.59000000 0.6500000 0.6476617 0.7150000 #> 16 good_er 2 sp 0.000000000 0.36000000 0.7300000 0.6476617 0.9400000 #> 17 good_er 2 sn 0.000000000 0.44000000 0.7300000 0.6476617 0.8600000 #> 18 good_er 2 prec 0.500000000 0.57894737 0.7300000 0.7409253 0.8888889 #> 19 good_er 2 mcc 0.070888121 0.26827335 0.3564168 0.3533310 0.4546786 #> 20 good_er 2 fscore 0.000000000 0.59060403 0.6931408 0.6110914 0.7063830 #> 21 good_er 3 rank 0.000000000 0.25000000 0.5000000 0.5000000 0.7500000 #> 22 good_er 3 score 0.001182349 0.11879598 0.2442350 0.3444894 0.5310979 #> 23 good_er 3 label -1.000000000 -1.00000000 0.0000000 0.0000000 1.0000000 #> 24 good_er 3 err 0.245000000 0.29000000 0.3400000 0.3503980 0.4100000 #> 25 good_er 3 acc 0.495000000 0.59000000 0.6600000 0.6496020 0.7100000 #> 26 good_er 3 sp 0.000000000 0.40000000 0.7100000 0.6496020 0.9700000 #> 27 good_er 3 sn 0.000000000 0.47000000 0.7100000 0.6496020 0.9000000 #> 28 good_er 3 prec 0.497461929 0.60000000 0.7100000 0.7510239 0.9400000 #> 29 good_er 3 mcc -0.070888121 0.27238140 0.3604768 0.3535845 0.4745113 #> 30 good_er 3 fscore 0.000000000 0.62666667 0.6951220 0.6154287 0.7093596 #> 31 good_er 4 rank 0.000000000 0.25000000 0.5000000 0.5000000 0.7500000 #> 32 good_er 4 score 0.002909921 0.09619691 0.2205083 0.3109069 0.5103813 #> 33 good_er 4 label -1.000000000 -1.00000000 0.0000000 0.0000000 1.0000000 #> 34 good_er 4 err 0.260000000 0.32500000 0.3550000 0.3691045 0.4150000 #> 35 good_er 4 acc 0.500000000 0.58500000 0.6450000 0.6308955 0.6750000 #> 36 good_er 4 sp 0.000000000 0.37000000 0.6600000 0.6308955 0.9800000 #> 37 good_er 4 sn 0.000000000 0.48000000 0.6600000 0.6308955 0.8700000 #> 38 good_er 4 prec 0.500000000 0.58169935 0.6601942 0.7298597 0.9523810 #> 39 good_er 4 mcc 0.070888121 0.26107554 0.3098970 0.3146643 0.3739102 #> 40 good_er 4 fscore 0.000000000 0.63576159 0.6666667 0.5964046 0.6844444 #> Max. #> 1 1.0000000 #> 2 0.9678504 #> 3 1.0000000 #> 4 0.5000000 #> 5 0.7600000 #> 6 1.0000000 #> 7 1.0000000 #> 8 1.0000000 #> 9 0.5286344 #> 10 0.7567568 #> 11 1.0000000 #> 12 0.9880810 #> 13 1.0000000 #> 14 0.5000000 #> 15 0.7600000 #> 16 1.0000000 #> 17 1.0000000 #> 18 1.0000000 #> 19 0.5226197 #> 20 0.7500000 #> 21 1.0000000 #> 22 0.9949425 #> 23 1.0000000 #> 24 0.5050000 #> 25 0.7550000 #> 26 1.0000000 #> 27 1.0000000 #> 28 1.0000000 #> 29 0.5444357 #> 30 0.7289720 #> 31 1.0000000 #> 32 0.9933312 #> 33 1.0000000 #> 34 0.5000000 #> 35 0.7400000 #> 36 1.0000000 #> 37 1.0000000 #> 38 1.0000000 #> 39 0.5345225 #> 40 0.7058824 #> #> #> === Input data === #> #> Model name Dataset ID # of negatives # of positives #> 1 good_er 1 100 100 #> 2 good_er 2 100 100 #> 3 good_er 3 100 100 #> 4 good_er 4 100 100 #>
################################################## ### Multiple models & multiple test datasets ### ## Create sample datasets with 100 positives and 100 negatives samps <- create_sim_samples(4, 100, 100, "all") mdat <- mmdata(samps[["scores"]], samps[["labels"]], modnames = samps[["modnames"]], dsids = samps[["dsids"]]) ## Generate an mmcurve object that contains ROC and Precision-Recall curves mmcurves <- evalmod(mdat) mmcurves
#> #> === AUCs === #> #> Model name Dataset ID Curve type AUC #> 1 random 1 ROC 0.4664000 #> 2 random 1 PRC 0.4554721 #> 3 poor_er 1 ROC 0.8413000 #> 4 poor_er 1 PRC 0.8323357 #> 5 good_er 1 ROC 0.7869000 #> 6 good_er 1 PRC 0.8235637 #> 7 excel 1 ROC 0.9747000 #> 8 excel 1 PRC 0.9770208 #> 9 perf 1 ROC 1.0000000 #> 10 perf 1 PRC 1.0000000 #> 11 random 2 ROC 0.5010000 #> 12 random 2 PRC 0.5297762 #> 13 poor_er 2 ROC 0.7978000 #> 14 poor_er 2 PRC 0.7599845 #> 15 good_er 2 ROC 0.8130000 #> 16 good_er 2 PRC 0.8507086 #> 17 excel 2 ROC 0.9820000 #> 18 excel 2 PRC 0.9818294 #> 19 perf 2 ROC 1.0000000 #> 20 perf 2 PRC 1.0000000 #> 21 random 3 ROC 0.4534000 #> 22 random 3 PRC 0.4995084 #> 23 poor_er 3 ROC 0.8173000 #> 24 poor_er 3 PRC 0.7708199 #> 25 good_er 3 ROC 0.8236000 #> 26 good_er 3 PRC 0.8589273 #> 27 excel 3 ROC 0.9792000 #> 28 excel 3 PRC 0.9798972 #> 29 perf 3 ROC 1.0000000 #> 30 perf 3 PRC 1.0000000 #> 31 random 4 ROC 0.5158000 #> 32 random 4 PRC 0.4979247 #> 33 poor_er 4 ROC 0.7891000 #> 34 poor_er 4 PRC 0.7582069 #> 35 good_er 4 ROC 0.7403000 #> 36 good_er 4 PRC 0.7848398 #> 37 excel 4 ROC 0.9917000 #> 38 excel 4 PRC 0.9918231 #> 39 perf 4 ROC 1.0000000 #> 40 perf 4 PRC 1.0000000 #> #> #> === Input data === #> #> Model name Dataset ID # of negatives # of positives #> 1 random 1 100 100 #> 2 poor_er 1 100 100 #> 3 good_er 1 100 100 #> 4 excel 1 100 100 #> 5 perf 1 100 100 #> 6 random 2 100 100 #> 7 poor_er 2 100 100 #> 8 good_er 2 100 100 #> 9 excel 2 100 100 #> 10 perf 2 100 100 #> 11 random 3 100 100 #> 12 poor_er 3 100 100 #> 13 good_er 3 100 100 #> 14 excel 3 100 100 #> 15 perf 3 100 100 #> 16 random 4 100 100 #> 17 poor_er 4 100 100 #> 18 good_er 4 100 100 #> 19 excel 4 100 100 #> 20 perf 4 100 100 #>
## Generate an mmpoints object that contains basic evaluation measures mmpoints <- evalmod(mdat, mode = "basic") mmpoints
#> #> === Basic performance evaluation measures === #> #> ## Performance measures (Meas.) #> rank: normalized rank #> score: score #> label: label #> err: error rate #> acc: accuracy #> sp: specificity #> sn: sensitivity #> prec: precision #> mcc: Matthews correlation coefficient #> fscore: F-score #> #> #> Model ID Meas. Min. 1st Qu. Median Mean #> 1 random 1 rank 0.0000000000 0.25000000 0.50000000 0.50000000 #> 2 random 1 score -3.7196447669 -0.72546621 0.01966020 0.03852809 #> 3 random 1 label -1.0000000000 -1.00000000 0.00000000 0.00000000 #> 4 random 1 err 0.4800000000 0.50500000 0.51500000 0.51671642 #> 5 random 1 acc 0.4350000000 0.47000000 0.48500000 0.48328358 #> 6 random 1 sp 0.0000000000 0.24000000 0.51000000 0.48328358 #> 7 random 1 sn 0.0000000000 0.23000000 0.51000000 0.48328358 #> 8 random 1 prec 0.0000000000 0.44680851 0.48780488 0.44037245 #> 9 random 1 mcc -0.1965415260 -0.07432715 -0.03686049 -0.04888954 #> 10 random 1 fscore 0.0000000000 0.30666667 0.50746269 0.42849335 #> 11 poor_er 1 rank 0.0000000000 0.25000000 0.50000000 0.50000000 #> 12 poor_er 1 score 0.0171065780 0.47810555 0.72400415 0.64598106 #> 13 poor_er 1 label -1.0000000000 -1.00000000 0.00000000 0.00000000 #> 14 poor_er 1 err 0.2350000000 0.26000000 0.31000000 0.33019900 #> 15 poor_er 1 acc 0.5000000000 0.61500000 0.69000000 0.66980100 #> 16 poor_er 1 sp 0.0000000000 0.47000000 0.75000000 0.66980100 #> 17 poor_er 1 sn 0.0000000000 0.43000000 0.75000000 0.66980100 #> 18 poor_er 1 prec 0.5000000000 0.64666667 0.75247525 0.75032741 #> 19 poor_er 1 mcc 0.0708881205 0.34791713 0.42759148 0.40531080 #> 20 poor_er 1 fscore 0.0000000000 0.57333333 0.70422535 0.62700334 #> 21 good_er 1 rank 0.0000000000 0.25000000 0.50000000 0.50000000 #> 22 good_er 1 score 0.0038531523 0.12479604 0.24937765 0.33448031 #> 23 good_er 1 label -1.0000000000 -1.00000000 0.00000000 0.00000000 #> 24 good_er 1 err 0.2400000000 0.28500000 0.34500000 0.35726368 #> 25 good_er 1 acc 0.4950000000 0.56500000 0.65500000 0.64273632 #> 26 good_er 1 sp 0.0000000000 0.37000000 0.75000000 0.64273632 #> 27 good_er 1 sn 0.0000000000 0.44000000 0.75000000 0.64273632 #> 28 good_er 1 prec 0.4974358974 0.58000000 0.75000000 0.73894780 #> 29 good_er 1 mcc -0.0320256308 0.21901763 0.37465377 0.33204549 #> 30 good_er 1 fscore 0.0000000000 0.58666667 0.67169811 0.60784609 #> 31 excel 1 rank 0.0000000000 0.25000000 0.50000000 0.50000000 #> 32 excel 1 score -2.4084888369 0.07990728 1.36014670 1.50931459 #> 33 excel 1 label -1.0000000000 -1.00000000 0.00000000 0.00000000 #> 34 excel 1 err 0.0900000000 0.13000000 0.25000000 0.26383085 #> 35 excel 1 acc 0.5000000000 0.62500000 0.75000000 0.73616915 #> 36 excel 1 sp 0.0000000000 0.50000000 0.88000000 0.73616915 #> 37 excel 1 sn 0.0000000000 0.50000000 0.88000000 0.73616915 #> 38 excel 1 prec 0.5000000000 0.66666667 0.87878788 0.83369566 #> 39 excel 1 mcc 0.0708881205 0.38226007 0.57735027 0.54793967 #> 40 excel 1 fscore 0.0000000000 0.66666667 0.76335878 0.69796621 #> 41 perf 1 rank 0.0000000000 0.25000000 0.50000000 0.50000000 #> 42 perf 1 score 0.0000000000 0.00000000 0.50000000 0.50000000 #> 43 perf 1 label -1.0000000000 -1.00000000 0.00000000 0.00000000 #> 44 perf 1 err 0.0000000000 0.12500000 0.25000000 0.25124378 #> 45 perf 1 acc 0.5000000000 0.62500000 0.75000000 0.74875622 #> 46 perf 1 sp 0.0000000000 0.50000000 1.00000000 0.74875622 #> 47 perf 1 sn 0.0000000000 0.50000000 1.00000000 0.74875622 #> 48 perf 1 prec 0.5000000000 0.66666667 1.00000000 0.84609623 #> 49 perf 1 mcc 0.0708881205 0.38226007 0.57735027 0.57352524 #> 50 perf 1 fscore 0.0000000000 0.66666667 0.76335878 0.71042736 #> 51 random 2 rank 0.0000000000 0.25000000 0.50000000 0.50000000 #> 52 random 2 score -2.7126690768 -0.61857552 0.03007718 0.03119488 #> 53 random 2 label -1.0000000000 -1.00000000 0.00000000 0.00000000 #> 54 random 2 err 0.4550000000 0.48000000 0.49500000 0.49950249 #> 55 random 2 acc 0.4350000000 0.48500000 0.50500000 0.50049751 #> 56 random 2 sp 0.0000000000 0.26000000 0.44000000 0.50049751 #> 57 random 2 sn 0.0000000000 0.27000000 0.44000000 0.50049751 #> 58 random 2 prec 0.4343434343 0.48000000 0.50724638 0.52430204 #> 59 random 2 mcc -0.1300065005 -0.03218007 0.02252458 0.01600532 #> 60 random 2 fscore 0.0000000000 0.35000000 0.44680851 0.45216266 #> 61 poor_er 2 rank 0.0000000000 0.25000000 0.50000000 0.50000000 #> 62 poor_er 2 score 0.0024159190 0.47923214 0.72985673 0.65997903 #> 63 poor_er 2 label -1.0000000000 -1.00000000 0.00000000 0.00000000 #> 64 poor_er 2 err 0.2650000000 0.29000000 0.32500000 0.35184080 #> 65 poor_er 2 acc 0.5000000000 0.59000000 0.67500000 0.64815920 #> 66 poor_er 2 sp 0.0000000000 0.45000000 0.71000000 0.64815920 #> 67 poor_er 2 sn 0.0000000000 0.38000000 0.71000000 0.64815920 #> 68 poor_er 2 prec 0.5000000000 0.63333333 0.71153846 0.70664393 #> 69 poor_er 2 mcc 0.0708881205 0.26741313 0.39834824 0.35375755 #> 70 poor_er 2 fscore 0.0000000000 0.50331126 0.69841270 0.60118108 #> 71 good_er 2 rank 0.0000000000 0.25000000 0.50000000 0.50000000 #> 72 good_er 2 score 0.0008120111 0.12826970 0.27148064 0.35228446 #> 73 good_er 2 label -1.0000000000 -1.00000000 0.00000000 0.00000000 #> 74 good_er 2 err 0.2350000000 0.27000000 0.33000000 0.34427861 #> 75 good_er 2 acc 0.4950000000 0.59000000 0.67000000 0.65572139 #> 76 good_er 2 sp 0.0000000000 0.38000000 0.75000000 0.65572139 #> 77 good_er 2 sn 0.0000000000 0.47000000 0.75000000 0.65572139 #> 78 good_er 2 prec 0.4974874372 0.58666667 0.75000000 0.75551150 #> 79 good_er 2 mcc -0.0708881205 0.26434061 0.40347329 0.36619580 #> 80 good_er 2 fscore 0.0000000000 0.62666667 0.69090909 0.62102015 #> 81 excel 2 rank 0.0000000000 0.25000000 0.50000000 0.50000000 #> 82 excel 2 score -2.1999085293 -0.12103461 1.60602909 1.58678161 #> 83 excel 2 label -1.0000000000 -1.00000000 0.00000000 0.00000000 #> 84 excel 2 err 0.0750000000 0.13500000 0.25000000 0.26019900 #> 85 excel 2 acc 0.5000000000 0.62500000 0.75000000 0.73980100 #> 86 excel 2 sp 0.0000000000 0.50000000 0.92000000 0.73980100 #> 87 excel 2 sn 0.0000000000 0.50000000 0.92000000 0.73980100 #> 88 excel 2 prec 0.5000000000 0.66666667 0.91919192 0.83664007 #> 89 excel 2 mcc 0.0708881205 0.38226007 0.57735027 0.55531505 #> 90 excel 2 fscore 0.0000000000 0.66666667 0.76335878 0.70126069 #> 91 perf 2 rank 0.0000000000 0.25000000 0.50000000 0.50000000 #> 92 perf 2 score 0.0000000000 0.00000000 0.50000000 0.50000000 #> 93 perf 2 label -1.0000000000 -1.00000000 0.00000000 0.00000000 #> 94 perf 2 err 0.0000000000 0.12500000 0.25000000 0.25124378 #> 95 perf 2 acc 0.5000000000 0.62500000 0.75000000 0.74875622 #> 96 perf 2 sp 0.0000000000 0.50000000 1.00000000 0.74875622 #> 97 perf 2 sn 0.0000000000 0.50000000 1.00000000 0.74875622 #> 98 perf 2 prec 0.5000000000 0.66666667 1.00000000 0.84609623 #> 99 perf 2 mcc 0.0708881205 0.38226007 0.57735027 0.57352524 #> 100 perf 2 fscore 0.0000000000 0.66666667 0.76335878 0.71042736 #> 101 random 3 rank 0.0000000000 0.25000000 0.50000000 0.50000000 #> 102 random 3 score -2.7448136984 -0.71403529 0.10433430 0.05253232 #> 103 random 3 label -1.0000000000 -1.00000000 0.00000000 0.00000000 #> 104 random 3 err 0.4800000000 0.51000000 0.52000000 0.52318408 #> 105 random 3 acc 0.4450000000 0.46500000 0.48000000 0.47681592 #> 106 random 3 sp 0.0000000000 0.21000000 0.45000000 0.47681592 #> 107 random 3 sn 0.0000000000 0.24000000 0.45000000 0.47681592 #> 108 random 3 prec 0.4062500000 0.46296296 0.47651007 0.49301710 #> 109 random 3 mcc -0.1428571429 -0.08647316 -0.06735267 -0.05445496 #> 110 random 3 fscore 0.0000000000 0.32000000 0.45000000 0.42853690 #> 111 poor_er 3 rank 0.0000000000 0.25000000 0.50000000 0.50000000 #> 112 poor_er 3 score 0.0079912755 0.43358938 0.69041993 0.62787909 #> 113 poor_er 3 label -1.0000000000 -1.00000000 0.00000000 0.00000000 #> 114 poor_er 3 err 0.2550000000 0.27500000 0.32000000 0.34213930 #> 115 poor_er 3 acc 0.5000000000 0.59500000 0.68000000 0.65786070 #> 116 poor_er 3 sp 0.0000000000 0.46000000 0.72000000 0.65786070 #> 117 poor_er 3 sn 0.0000000000 0.40000000 0.72000000 0.65786070 #> 118 poor_er 3 prec 0.5000000000 0.64052288 0.71962617 0.71466253 #> 119 poor_er 3 mcc 0.0411345035 0.28257220 0.42828957 0.37389651 #> 120 poor_er 3 fscore 0.0000000000 0.53333333 0.70329670 0.61159471 #> 121 good_er 3 rank 0.0000000000 0.25000000 0.50000000 0.50000000 #> 122 good_er 3 score 0.0002204991 0.11651908 0.26319338 0.34018141 #> 123 good_er 3 label -1.0000000000 -1.00000000 0.00000000 0.00000000 #> 124 good_er 3 err 0.2450000000 0.26500000 0.32000000 0.33900498 #> 125 good_er 3 acc 0.5000000000 0.60000000 0.68000000 0.66099502 #> 126 good_er 3 sp 0.0000000000 0.37000000 0.74000000 0.66099502 #> 127 good_er 3 sn 0.0000000000 0.48000000 0.74000000 0.66099502 #> 128 good_er 3 prec 0.5000000000 0.58389262 0.73786408 0.76077342 #> 129 good_er 3 mcc 0.0000000000 0.28916343 0.43124459 0.38019576 #> 130 good_er 3 fscore 0.0000000000 0.63157895 0.69803922 0.62578629 #> 131 excel 3 rank 0.0000000000 0.25000000 0.50000000 0.50000000 #> 132 excel 3 score -3.1864675050 -0.10870387 1.37845182 1.28729428 #> 133 excel 3 label -1.0000000000 -1.00000000 0.00000000 0.00000000 #> 134 excel 3 err 0.0650000000 0.14000000 0.25000000 0.26159204 #> 135 excel 3 acc 0.5000000000 0.62500000 0.75000000 0.73840796 #> 136 excel 3 sp 0.0000000000 0.50000000 0.91000000 0.73840796 #> 137 excel 3 sn 0.0000000000 0.50000000 0.91000000 0.73840796 #> 138 excel 3 prec 0.5000000000 0.66666667 0.91089109 0.83545580 #> 139 excel 3 mcc 0.0708881205 0.38226007 0.57735027 0.55238879 #> 140 excel 3 fscore 0.0000000000 0.66666667 0.76335878 0.69999345 #> 141 perf 3 rank 0.0000000000 0.25000000 0.50000000 0.50000000 #> 142 perf 3 score 0.0000000000 0.00000000 0.50000000 0.50000000 #> 143 perf 3 label -1.0000000000 -1.00000000 0.00000000 0.00000000 #> 144 perf 3 err 0.0000000000 0.12500000 0.25000000 0.25124378 #> 145 perf 3 acc 0.5000000000 0.62500000 0.75000000 0.74875622 #> 146 perf 3 sp 0.0000000000 0.50000000 1.00000000 0.74875622 #> 147 perf 3 sn 0.0000000000 0.50000000 1.00000000 0.74875622 #> 148 perf 3 prec 0.5000000000 0.66666667 1.00000000 0.84609623 #> 149 perf 3 mcc 0.0708881205 0.38226007 0.57735027 0.57352524 #> 150 perf 3 fscore 0.0000000000 0.66666667 0.76335878 0.71042736 #> 151 random 4 rank 0.0000000000 0.25000000 0.50000000 0.50000000 #> 152 random 4 score -2.7257999419 -0.67296831 -0.01597703 0.09431798 #> 153 random 4 label -1.0000000000 -1.00000000 0.00000000 0.00000000 #> 154 random 4 err 0.4600000000 0.47500000 0.49000000 0.49213930 #> 155 random 4 acc 0.4750000000 0.49500000 0.51000000 0.50786070 #> 156 random 4 sp 0.0000000000 0.23000000 0.53000000 0.50786070 #> 157 random 4 sn 0.0000000000 0.25000000 0.53000000 0.50786070 #> 158 random 4 prec 0.0000000000 0.49275362 0.50757576 0.49209950 #> 159 random 4 mcc -0.1014092393 -0.02376690 0.02060214 0.01463156 #> 160 random 4 fscore 0.0000000000 0.33333333 0.52475248 0.45672938 #> 161 poor_er 4 rank 0.0000000000 0.25000000 0.50000000 0.50000000 #> 162 poor_er 4 score 0.0263964448 0.46654176 0.74462004 0.66850804 #> 163 poor_er 4 label -1.0000000000 -1.00000000 0.00000000 0.00000000 #> 164 poor_er 4 err 0.2750000000 0.31000000 0.33000000 0.35616915 #> 165 poor_er 4 acc 0.5000000000 0.60000000 0.67000000 0.64383085 #> 166 poor_er 4 sp 0.0000000000 0.45000000 0.69000000 0.64383085 #> 167 poor_er 4 sn 0.0000000000 0.39000000 0.69000000 0.64383085 #> 168 poor_er 4 prec 0.5000000000 0.63157895 0.68817204 0.70481202 #> 169 poor_er 4 mcc 0.0708881205 0.29552426 0.37046337 0.34550181 #> 170 poor_er 4 fscore 0.0000000000 0.52000000 0.67692308 0.59766494 #> 171 good_er 4 rank 0.0000000000 0.25000000 0.50000000 0.50000000 #> 172 good_er 4 score 0.0018165403 0.12029054 0.28204248 0.34952361 #> 173 good_er 4 label -1.0000000000 -1.00000000 0.00000000 0.00000000 #> 174 good_er 4 err 0.3150000000 0.33000000 0.37000000 0.38044776 #> 175 good_er 4 acc 0.4950000000 0.58500000 0.63000000 0.61955224 #> 176 good_er 4 sp 0.0000000000 0.35000000 0.68000000 0.61955224 #> 177 good_er 4 sn 0.0000000000 0.42000000 0.68000000 0.61955224 #> 178 good_er 4 prec 0.4974874372 0.57051282 0.67368421 0.71275940 #> 179 good_er 4 mcc -0.0708881205 0.24311921 0.31265318 0.28636990 #> 180 good_er 4 fscore 0.0000000000 0.56375839 0.66442953 0.58330745 #> 181 excel 4 rank 0.0000000000 0.25000000 0.50000000 0.50000000 #> 182 excel 4 score -3.9822443515 -0.03772416 1.49055404 1.52605177 #> 183 excel 4 label -1.0000000000 -1.00000000 0.00000000 0.00000000 #> 184 excel 4 err 0.0350000000 0.12500000 0.25000000 0.25537313 #> 185 excel 4 acc 0.5000000000 0.62500000 0.75000000 0.74462687 #> 186 excel 4 sp 0.0000000000 0.50000000 0.95000000 0.74462687 #> 187 excel 4 sn 0.0000000000 0.50000000 0.95000000 0.74462687 #> 188 excel 4 prec 0.5000000000 0.66666667 0.95145631 0.84186856 #> 189 excel 4 mcc 0.0708881205 0.38226007 0.57735027 0.56514358 #> 190 excel 4 fscore 0.0000000000 0.66666667 0.76335878 0.70625948 #> 191 perf 4 rank 0.0000000000 0.25000000 0.50000000 0.50000000 #> 192 perf 4 score 0.0000000000 0.00000000 0.50000000 0.50000000 #> 193 perf 4 label -1.0000000000 -1.00000000 0.00000000 0.00000000 #> 194 perf 4 err 0.0000000000 0.12500000 0.25000000 0.25124378 #> 195 perf 4 acc 0.5000000000 0.62500000 0.75000000 0.74875622 #> 196 perf 4 sp 0.0000000000 0.50000000 1.00000000 0.74875622 #> 197 perf 4 sn 0.0000000000 0.50000000 1.00000000 0.74875622 #> 198 perf 4 prec 0.5000000000 0.66666667 1.00000000 0.84609623 #> 199 perf 4 mcc 0.0708881205 0.38226007 0.57735027 0.57352524 #> 200 perf 4 fscore 0.0000000000 0.66666667 0.76335878 0.71042736 #> 3rd Qu. Max. #> 1 0.75000000 1.00000000 #> 2 0.70055951 3.72459045 #> 3 1.00000000 1.00000000 #> 4 0.53000000 0.56500000 #> 5 0.49500000 0.52000000 #> 6 0.73000000 1.00000000 #> 7 0.74000000 1.00000000 #> 8 0.49629630 0.51818182 #> 9 -0.01081192 0.04020151 #> 10 0.59200000 0.66666667 #> 11 0.75000000 1.00000000 #> 12 0.86815165 0.99912740 #> 13 1.00000000 1.00000000 #> 14 0.38500000 0.50000000 #> 15 0.74000000 0.76500000 #> 16 0.93000000 1.00000000 #> 17 0.97000000 1.00000000 #> 18 0.85000000 1.00000000 #> 19 0.50181801 0.54460039 #> 20 0.75968992 0.78923767 #> 21 0.75000000 1.00000000 #> 22 0.49585506 0.98175221 #> 23 1.00000000 1.00000000 #> 24 0.43500000 0.50500000 #> 25 0.71500000 0.76000000 #> 26 0.94000000 1.00000000 #> 27 0.87000000 1.00000000 #> 28 0.88095238 1.00000000 #> 29 0.46367456 0.52378493 #> 30 0.71748879 0.75728155 #> 31 0.75000000 1.00000000 #> 32 3.02830574 5.00922214 #> 33 1.00000000 1.00000000 #> 34 0.37500000 0.50000000 #> 35 0.87000000 0.91000000 #> 36 1.00000000 1.00000000 #> 37 1.00000000 1.00000000 #> 38 1.00000000 1.00000000 #> 39 0.75154679 0.82815609 #> 40 0.87005650 0.90825688 #> 41 0.75000000 1.00000000 #> 42 1.00000000 1.00000000 #> 43 1.00000000 1.00000000 #> 44 0.37500000 0.50000000 #> 45 0.87500000 1.00000000 #> 46 1.00000000 1.00000000 #> 47 1.00000000 1.00000000 #> 48 1.00000000 1.00000000 #> 49 0.77459667 1.00000000 #> 50 0.87640449 1.00000000 #> 51 0.75000000 1.00000000 #> 52 0.63045676 2.38816297 #> 53 1.00000000 1.00000000 #> 54 0.51500000 0.56500000 #> 55 0.52000000 0.54500000 #> 56 0.77000000 1.00000000 #> 57 0.76000000 1.00000000 #> 58 0.52083333 1.00000000 #> 59 0.06948231 0.19044535 #> 60 0.60629921 0.68259386 #> 61 0.75000000 1.00000000 #> 62 0.89879002 0.99624627 #> 63 1.00000000 1.00000000 #> 64 0.41000000 0.50000000 #> 65 0.71000000 0.73500000 #> 66 0.88000000 1.00000000 #> 67 0.95000000 1.00000000 #> 68 0.77142857 1.00000000 #> 69 0.44489270 0.49217479 #> 70 0.74458874 0.76679842 #> 71 0.75000000 1.00000000 #> 72 0.53510009 0.99994002 #> 73 1.00000000 1.00000000 #> 74 0.41000000 0.50500000 #> 75 0.73000000 0.76500000 #> 76 0.97000000 1.00000000 #> 77 0.88000000 1.00000000 #> 78 0.93478261 1.00000000 #> 79 0.48996123 0.54554473 #> 80 0.72573840 0.75897436 #> 81 0.75000000 1.00000000 #> 82 3.25446026 5.06118069 #> 83 1.00000000 1.00000000 #> 84 0.37500000 0.50000000 #> 85 0.86500000 0.92500000 #> 86 1.00000000 1.00000000 #> 87 1.00000000 1.00000000 #> 88 1.00000000 1.00000000 #> 89 0.75604925 0.85518964 #> 90 0.86666667 0.92890995 #> 91 0.75000000 1.00000000 #> 92 1.00000000 1.00000000 #> 93 1.00000000 1.00000000 #> 94 0.37500000 0.50000000 #> 95 0.87500000 1.00000000 #> 96 1.00000000 1.00000000 #> 97 1.00000000 1.00000000 #> 98 1.00000000 1.00000000 #> 99 0.77459667 1.00000000 #> 100 0.87640449 1.00000000 #> 101 0.75000000 1.00000000 #> 102 0.89017956 2.23605269 #> 103 1.00000000 1.00000000 #> 104 0.53500000 0.55500000 #> 105 0.49000000 0.52000000 #> 106 0.74000000 1.00000000 #> 107 0.71000000 1.00000000 #> 108 0.48717949 1.00000000 #> 109 -0.03125213 0.14285714 #> 110 0.57370518 0.66666667 #> 111 0.75000000 1.00000000 #> 112 0.86882712 0.99993925 #> 113 1.00000000 1.00000000 #> 114 0.40500000 0.50000000 #> 115 0.72500000 0.74500000 #> 116 0.90000000 1.00000000 #> 117 0.96000000 1.00000000 #> 118 0.80000000 1.00000000 #> 119 0.47393590 0.51449576 #> 120 0.75117371 0.77966102 #> 121 0.75000000 1.00000000 #> 122 0.51572317 0.98204846 #> 123 1.00000000 1.00000000 #> 124 0.40000000 0.50000000 #> 125 0.73500000 0.75500000 #> 126 0.98000000 1.00000000 #> 127 0.87000000 1.00000000 #> 128 0.95652174 1.00000000 #> 129 0.49359270 0.54895863 #> 130 0.72268908 0.77419355 #> 131 0.75000000 1.00000000 #> 132 2.71190675 5.02411252 #> 133 1.00000000 1.00000000 #> 134 0.37500000 0.50000000 #> 135 0.86000000 0.93500000 #> 136 1.00000000 1.00000000 #> 137 1.00000000 1.00000000 #> 138 1.00000000 1.00000000 #> 139 0.73988011 0.87213937 #> 140 0.85714286 0.93264249 #> 141 0.75000000 1.00000000 #> 142 1.00000000 1.00000000 #> 143 1.00000000 1.00000000 #> 144 0.37500000 0.50000000 #> 145 0.87500000 1.00000000 #> 146 1.00000000 1.00000000 #> 147 1.00000000 1.00000000 #> 148 1.00000000 1.00000000 #> 149 0.77459667 1.00000000 #> 150 0.87640449 1.00000000 #> 151 0.75000000 1.00000000 #> 152 0.87150750 3.55247930 #> 153 1.00000000 1.00000000 #> 154 0.50500000 0.52500000 #> 155 0.52500000 0.54000000 #> 156 0.75000000 1.00000000 #> 157 0.73000000 1.00000000 #> 158 0.52100840 0.55932203 #> 159 0.05225971 0.11757927 #> 160 0.59055118 0.67132867 #> 161 0.75000000 1.00000000 #> 162 0.89913323 0.99595761 #> 163 1.00000000 1.00000000 #> 164 0.40000000 0.50000000 #> 165 0.69000000 0.72500000 #> 166 0.89000000 1.00000000 #> 167 0.95000000 1.00000000 #> 168 0.78000000 1.00000000 #> 169 0.42571468 0.48483561 #> 170 0.74074074 0.76859504 #> 171 0.75000000 1.00000000 #> 172 0.52477518 0.99723880 #> 173 1.00000000 1.00000000 #> 174 0.41500000 0.50500000 #> 175 0.67000000 0.68500000 #> 176 0.92000000 1.00000000 #> 177 0.85000000 1.00000000 #> 178 0.84615385 1.00000000 #> 179 0.36167334 0.42597488 #> 180 0.68000000 0.69961977 #> 181 0.75000000 1.00000000 #> 182 3.13379353 5.89375227 #> 183 1.00000000 1.00000000 #> 184 0.37500000 0.50000000 #> 185 0.87500000 0.96500000 #> 186 1.00000000 1.00000000 #> 187 1.00000000 1.00000000 #> 188 1.00000000 1.00000000 #> 189 0.77459667 0.93041878 #> 190 0.87150838 0.96551724 #> 191 0.75000000 1.00000000 #> 192 1.00000000 1.00000000 #> 193 1.00000000 1.00000000 #> 194 0.37500000 0.50000000 #> 195 0.87500000 1.00000000 #> 196 1.00000000 1.00000000 #> 197 1.00000000 1.00000000 #> 198 1.00000000 1.00000000 #> 199 0.77459667 1.00000000 #> 200 0.87640449 1.00000000 #> #> #> === Input data === #> #> Model name Dataset ID # of negatives # of positives #> 1 random 1 100 100 #> 2 poor_er 1 100 100 #> 3 good_er 1 100 100 #> 4 excel 1 100 100 #> 5 perf 1 100 100 #> 6 random 2 100 100 #> 7 poor_er 2 100 100 #> 8 good_er 2 100 100 #> 9 excel 2 100 100 #> 10 perf 2 100 100 #> 11 random 3 100 100 #> 12 poor_er 3 100 100 #> 13 good_er 3 100 100 #> 14 excel 3 100 100 #> 15 perf 3 100 100 #> 16 random 4 100 100 #> 17 poor_er 4 100 100 #> 18 good_er 4 100 100 #> 19 excel 4 100 100 #> 20 perf 4 100 100 #>
################################################## ### N-fold cross validation datasets ### ## Load test data data(M2N50F5) ## Speficy nessesary columns to create mdat cvdat <- mmdata(nfold_df = M2N50F5, score_cols = c(1, 2), lab_col = 3, fold_col = 4, modnames = c("m1", "m2"), dsids = 1:5) ## Generate an mmcurve object that contains ROC and Precision-Recall curves cvcurves <- evalmod(cvdat) cvcurves
#> #> === AUCs === #> #> Model name Dataset ID Curve type AUC #> 1 m1 1 ROC 1.0000000 #> 2 m1 1 PRC 1.0000000 #> 3 m1 2 ROC 0.4166667 #> 4 m1 2 PRC 0.5164199 #> 5 m1 3 ROC 0.2000000 #> 6 m1 3 PRC 0.4891743 #> 7 m1 4 ROC 0.7916667 #> 8 m1 4 PRC 0.7728152 #> 9 m1 5 ROC 0.4400000 #> 10 m1 5 PRC 0.4266312 #> 11 m2 1 ROC 0.4000000 #> 12 m2 1 PRC 0.4247188 #> 13 m2 2 ROC 0.7083333 #> 14 m2 2 PRC 0.6568625 #> 15 m2 3 ROC 0.8400000 #> 16 m2 3 PRC 0.9057736 #> 17 m2 4 ROC 0.7916667 #> 18 m2 4 PRC 0.8527712 #> 19 m2 5 ROC 0.4000000 #> 20 m2 5 PRC 0.4247188 #> #> #> === Input data === #> #> Model name Dataset ID # of negatives # of positives #> 1 m1 1 5 5 #> 2 m1 2 4 6 #> 3 m1 3 5 5 #> 4 m1 4 6 4 #> 5 m1 5 5 5 #> 6 m2 1 5 5 #> 7 m2 2 4 6 #> 8 m2 3 5 5 #> 9 m2 4 6 4 #> 10 m2 5 5 5 #>
## Generate an mmpoints object that contains basic evaluation measures cvpoints <- evalmod(cvdat, mode = "basic") cvpoints
#> #> === Basic performance evaluation measures === #> #> ## Performance measures (Meas.) #> rank: normalized rank #> score: score #> label: label #> err: error rate #> acc: accuracy #> sp: specificity #> sn: sensitivity #> prec: precision #> mcc: Matthews correlation coefficient #> fscore: F-score #> #> #> Model ID Meas. Min. 1st Qu. Median Mean #> 1 m1 1 rank 0.0000000 0.25000000 0.500000000 0.500000000 #> 2 m1 1 score -1.5761733 -0.92376396 -0.002327284 0.113500523 #> 3 m1 1 label -1.0000000 -1.00000000 0.000000000 0.000000000 #> 4 m1 1 err 0.0000000 0.15000000 0.300000000 0.272727273 #> 5 m1 1 acc 0.5000000 0.60000000 0.700000000 0.727272727 #> 6 m1 1 sp 0.0000000 0.50000000 1.000000000 0.727272727 #> 7 m1 1 sn 0.0000000 0.50000000 1.000000000 0.727272727 #> 8 m1 1 prec 0.5000000 0.66964286 1.000000000 0.838924964 #> 9 m1 1 mcc 0.3333333 0.50000000 0.654653671 0.623218574 #> 10 m1 1 fscore 0.0000000 0.61904762 0.750000000 0.676023471 #> 11 m1 2 rank 0.0000000 0.25000000 0.500000000 0.500000000 #> 12 m1 2 score -1.8425134 -1.08765052 -0.450813882 -0.375652956 #> 13 m1 2 label -1.0000000 -1.00000000 1.000000000 0.200000000 #> 14 m1 2 err 0.4000000 0.45000000 0.500000000 0.536363636 #> 15 m1 2 acc 0.3000000 0.40000000 0.500000000 0.463636364 #> 16 m1 2 sp 0.0000000 0.25000000 0.500000000 0.454545455 #> 17 m1 2 sn 0.0000000 0.16666667 0.500000000 0.469696970 #> 18 m1 2 prec 0.0000000 0.41666667 0.555555556 0.450180375 #> 19 m1 2 mcc -0.4082483 -0.27216553 -0.102062073 -0.125094358 #> 20 m1 2 fscore 0.0000000 0.23611111 0.545454545 0.439152766 #> 21 m1 3 rank 0.0000000 0.25000000 0.500000000 0.500000000 #> 22 m1 3 score -1.1558752 -0.16251342 0.389025335 0.414373732 #> 23 m1 3 label -1.0000000 -1.00000000 0.000000000 0.000000000 #> 24 m1 3 err 0.4000000 0.50000000 0.600000000 0.636363636 #> 25 m1 3 acc 0.1000000 0.25000000 0.400000000 0.363636364 #> 26 m1 3 sp 0.0000000 0.00000000 0.200000000 0.363636364 #> 27 m1 3 sn 0.0000000 0.20000000 0.200000000 0.363636364 #> 28 m1 3 prec 0.1666667 0.26785714 0.375000000 0.459559885 #> 29 m1 3 mcc -0.8164966 -0.60000000 -0.408248290 -0.355290715 #> 30 m1 3 fscore 0.0000000 0.21111111 0.285714286 0.318732278 #> 31 m1 4 rank 0.0000000 0.25000000 0.500000000 0.500000000 #> 32 m1 4 score -1.5462252 -1.19762686 -0.400691531 -0.141998509 #> 33 m1 4 label -1.0000000 -1.00000000 -1.000000000 -0.200000000 #> 34 m1 4 err 0.2000000 0.30000000 0.400000000 0.372727273 #> 35 m1 4 acc 0.4000000 0.60000000 0.600000000 0.627272727 #> 36 m1 4 sp 0.0000000 0.41666667 0.666666667 0.606060606 #> 37 m1 4 sn 0.0000000 0.50000000 0.750000000 0.659090909 #> 38 m1 4 prec 0.4000000 0.50000000 0.571428571 0.652958153 #> 39 m1 4 mcc 0.1666667 0.27216553 0.408248290 0.379646701 #> 40 m1 4 fscore 0.0000000 0.53571429 0.600000000 0.544137681 #> 41 m1 5 rank 0.0000000 0.25000000 0.500000000 0.500000000 #> 42 m1 5 score -1.5944199 -0.50932491 -0.113562820 0.006988569 #> 43 m1 5 label -1.0000000 -1.00000000 0.000000000 0.000000000 #> 44 m1 5 err 0.4000000 0.50000000 0.500000000 0.527272727 #> 45 m1 5 acc 0.3000000 0.40000000 0.500000000 0.472727273 #> 46 m1 5 sp 0.0000000 0.30000000 0.400000000 0.472727273 #> 47 m1 5 sn 0.0000000 0.10000000 0.400000000 0.472727273 #> 48 m1 5 prec 0.0000000 0.16666667 0.500000000 0.350937951 #> 49 m1 5 mcc -0.5000000 -0.21821789 0.000000000 -0.077777778 #> 50 m1 5 fscore 0.0000000 0.12500000 0.444444444 0.391172968 #> 51 m2 1 rank 0.0000000 0.25000000 0.500000000 0.500000000 #> 52 m2 1 score -1.5666375 -0.93714170 -0.338617003 -0.081054576 #> 53 m2 1 label -1.0000000 -1.00000000 0.000000000 0.000000000 #> 54 m2 1 err 0.5000000 0.50000000 0.500000000 0.545454545 #> 55 m2 1 acc 0.4000000 0.40000000 0.500000000 0.454545455 #> 56 m2 1 sp 0.0000000 0.20000000 0.400000000 0.454545455 #> 57 m2 1 sn 0.0000000 0.20000000 0.400000000 0.454545455 #> 58 m2 1 prec 0.0000000 0.36666667 0.444444444 0.373304473 #> 59 m2 1 mcc -0.3333333 -0.21821789 -0.200000000 -0.144789161 #> 60 m2 1 fscore 0.0000000 0.26785714 0.444444444 0.389008466 #> 61 m2 2 rank 0.0000000 0.25000000 0.500000000 0.500000000 #> 62 m2 2 score -0.9422955 -0.51338997 0.168580899 -0.003990330 #> 63 m2 2 label -1.0000000 -1.00000000 1.000000000 0.200000000 #> 64 m2 2 err 0.2000000 0.30000000 0.400000000 0.409090909 #> 65 m2 2 acc 0.3000000 0.45000000 0.600000000 0.590909091 #> 66 m2 2 sp 0.0000000 0.50000000 0.750000000 0.613636364 #> 67 m2 2 sn 0.0000000 0.25000000 0.666666667 0.575757576 #> 68 m2 2 prec 0.0000000 0.55000000 0.666666667 0.570995671 #> 69 m2 2 mcc -0.4082483 0.08908708 0.356348323 0.244147488 #> 70 m2 2 fscore 0.0000000 0.34722222 0.727272727 0.548311285 #> 71 m2 3 rank 0.0000000 0.25000000 0.500000000 0.500000000 #> 72 m2 3 score -1.0962763 -0.40057827 -0.167608702 0.082174350 #> 73 m2 3 label -1.0000000 -1.00000000 0.000000000 0.000000000 #> 74 m2 3 err 0.1000000 0.25000000 0.400000000 0.345454545 #> 75 m2 3 acc 0.5000000 0.55000000 0.600000000 0.654545455 #> 76 m2 3 sp 0.0000000 0.30000000 0.800000000 0.654545455 #> 77 m2 3 sn 0.0000000 0.50000000 0.800000000 0.654545455 #> 78 m2 3 prec 0.5000000 0.56349206 0.800000000 0.781240981 #> 79 m2 3 mcc 0.0000000 0.33333333 0.408248290 0.429364789 #> 80 m2 3 fscore 0.0000000 0.59340659 0.666666667 0.612175199 #> 81 m2 4 rank 0.0000000 0.25000000 0.500000000 0.500000000 #> 82 m2 4 score -1.1654721 -0.35410360 -0.099106516 0.103674375 #> 83 m2 4 label -1.0000000 -1.00000000 -1.000000000 -0.200000000 #> 84 m2 4 err 0.1000000 0.25000000 0.400000000 0.372727273 #> 85 m2 4 acc 0.4000000 0.50000000 0.600000000 0.627272727 #> 86 m2 4 sp 0.0000000 0.25000000 0.666666667 0.606060606 #> 87 m2 4 sn 0.0000000 0.62500000 0.750000000 0.659090909 #> 88 m2 4 prec 0.3750000 0.43650794 0.600000000 0.681637807 #> 89 m2 4 mcc -0.1020621 0.25000000 0.408248290 0.369241846 #> 90 m2 4 fscore 0.0000000 0.52272727 0.600000000 0.561158538 #> 91 m2 5 rank 0.0000000 0.25000000 0.500000000 0.500000000 #> 92 m2 5 score -1.7908147 -1.14209817 0.032207934 -0.329061088 #> 93 m2 5 label -1.0000000 -1.00000000 0.000000000 0.000000000 #> 94 m2 5 err 0.5000000 0.50000000 0.500000000 0.545454545 #> 95 m2 5 acc 0.4000000 0.40000000 0.500000000 0.454545455 #> 96 m2 5 sp 0.0000000 0.20000000 0.400000000 0.454545455 #> 97 m2 5 sn 0.0000000 0.20000000 0.400000000 0.454545455 #> 98 m2 5 prec 0.0000000 0.36666667 0.444444444 0.373304473 #> 99 m2 5 mcc -0.3333333 -0.21821789 -0.200000000 -0.144789161 #> 100 m2 5 fscore 0.0000000 0.26785714 0.444444444 0.389008466 #> 3rd Qu. Max. #> 1 0.75000000 1.0000000 #> 2 1.06708368 2.0606025 #> 3 1.00000000 1.0000000 #> 4 0.40000000 0.5000000 #> 5 0.85000000 1.0000000 #> 6 1.00000000 1.0000000 #> 7 1.00000000 1.0000000 #> 8 1.00000000 1.0000000 #> 9 0.81649658 1.0000000 #> 10 0.86111111 1.0000000 #> 11 0.75000000 1.0000000 #> 12 -0.08830171 2.2002973 #> 13 1.00000000 1.0000000 #> 14 0.60000000 0.7000000 #> 15 0.55000000 0.6000000 #> 16 0.62500000 1.0000000 #> 17 0.75000000 1.0000000 #> 18 0.60000000 0.6666667 #> 19 0.00000000 0.1666667 #> 20 0.66666667 0.7500000 #> 21 0.75000000 1.0000000 #> 22 1.01753386 1.6870336 #> 23 1.00000000 1.0000000 #> 24 0.75000000 0.9000000 #> 25 0.50000000 0.6000000 #> 26 0.70000000 1.0000000 #> 27 0.50000000 1.0000000 #> 28 0.50000000 1.0000000 #> 29 -0.21821789 0.3333333 #> 30 0.39743590 0.6666667 #> 31 0.75000000 1.0000000 #> 32 1.00867637 1.5387145 #> 33 1.00000000 1.0000000 #> 34 0.40000000 0.6000000 #> 35 0.70000000 0.8000000 #> 36 0.91666667 1.0000000 #> 37 1.00000000 1.0000000 #> 38 0.83333333 1.0000000 #> 39 0.40824829 0.6123724 #> 40 0.66666667 0.7272727 #> 41 0.75000000 1.0000000 #> 42 0.45270677 1.9237289 #> 43 1.00000000 1.0000000 #> 44 0.60000000 0.7000000 #> 45 0.50000000 0.6000000 #> 46 0.60000000 1.0000000 #> 47 0.80000000 1.0000000 #> 48 0.50000000 0.5714286 #> 49 0.00000000 0.3333333 #> 50 0.64102564 0.7142857 #> 51 0.75000000 1.0000000 #> 52 0.43266892 2.8510651 #> 53 1.00000000 1.0000000 #> 54 0.60000000 0.6000000 #> 55 0.50000000 0.5000000 #> 56 0.70000000 1.0000000 #> 57 0.70000000 1.0000000 #> 58 0.50000000 0.5000000 #> 59 0.00000000 0.0000000 #> 60 0.55844156 0.6666667 #> 61 0.75000000 1.0000000 #> 62 0.39797893 0.7890534 #> 63 1.00000000 1.0000000 #> 64 0.55000000 0.7000000 #> 65 0.70000000 0.8000000 #> 66 0.75000000 1.0000000 #> 67 0.91666667 1.0000000 #> 68 0.75000000 0.8333333 #> 69 0.40824829 0.6123724 #> 70 0.78461538 0.8571429 #> 71 0.75000000 1.0000000 #> 72 0.35464019 2.0647833 #> 73 1.00000000 1.0000000 #> 74 0.45000000 0.5000000 #> 75 0.75000000 0.9000000 #> 76 1.00000000 1.0000000 #> 77 0.80000000 1.0000000 #> 78 1.00000000 1.0000000 #> 79 0.60000000 0.8164966 #> 80 0.73863636 0.8888889 #> 81 0.75000000 1.0000000 #> 82 0.79072036 1.4431452 #> 83 1.00000000 1.0000000 #> 84 0.50000000 0.6000000 #> 85 0.75000000 0.9000000 #> 86 1.00000000 1.0000000 #> 87 0.75000000 1.0000000 #> 88 1.00000000 1.0000000 #> 89 0.58333333 0.8017837 #> 90 0.66666667 0.8571429 #> 91 0.75000000 1.0000000 #> 92 0.31323748 0.7002948 #> 93 1.00000000 1.0000000 #> 94 0.60000000 0.6000000 #> 95 0.50000000 0.5000000 #> 96 0.70000000 1.0000000 #> 97 0.70000000 1.0000000 #> 98 0.50000000 0.5000000 #> 99 0.00000000 0.0000000 #> 100 0.55844156 0.6666667 #> #> #> === Input data === #> #> Model name Dataset ID # of negatives # of positives #> 1 m1 1 5 5 #> 2 m1 2 4 6 #> 3 m1 3 5 5 #> 4 m1 4 6 4 #> 5 m1 5 5 5 #> 6 m2 1 5 5 #> 7 m2 2 4 6 #> 8 m2 3 5 5 #> 9 m2 4 6 4 #> 10 m2 5 5 5 #>
## Specify mmdata arguments from evalmod cvcurves2 <- evalmod(nfold_df = M2N50F5, score_cols = c(1, 2), lab_col = 3, fold_col = 4, modnames = c("m1", "m2"), dsids = 1:5) cvcurves2
#> #> === AUCs === #> #> Model name Dataset ID Curve type AUC #> 1 m1 1 ROC 1.0000000 #> 2 m1 1 PRC 1.0000000 #> 3 m1 2 ROC 0.4166667 #> 4 m1 2 PRC 0.5164199 #> 5 m1 3 ROC 0.2000000 #> 6 m1 3 PRC 0.4891743 #> 7 m1 4 ROC 0.7916667 #> 8 m1 4 PRC 0.7728152 #> 9 m1 5 ROC 0.4400000 #> 10 m1 5 PRC 0.4266312 #> 11 m2 1 ROC 0.4000000 #> 12 m2 1 PRC 0.4247188 #> 13 m2 2 ROC 0.7083333 #> 14 m2 2 PRC 0.6568625 #> 15 m2 3 ROC 0.8400000 #> 16 m2 3 PRC 0.9057736 #> 17 m2 4 ROC 0.7916667 #> 18 m2 4 PRC 0.8527712 #> 19 m2 5 ROC 0.4000000 #> 20 m2 5 PRC 0.4247188 #> #> #> === Input data === #> #> Model name Dataset ID # of negatives # of positives #> 1 m1 1 5 5 #> 2 m1 2 4 6 #> 3 m1 3 5 5 #> 4 m1 4 6 4 #> 5 m1 5 5 5 #> 6 m2 1 5 5 #> 7 m2 2 4 6 #> 8 m2 3 5 5 #> 9 m2 4 6 4 #> 10 m2 5 5 5 #>
################################################## ### AUC with the U statistic ### ## mode = "aucroc" returns 'aucroc' S3 object data(P10N10) # 'aucroc' S3 object uauc1 <- evalmod(scores = P10N10$scores, labels = P10N10$labels, mode="aucroc") # print 'aucroc' uauc1
#> #> === Input data === #> #> Model name Dataset ID # of negatives # of positives #> 1 m1 1 10 10 #> #> #> === AUCs === #> #> Model name Dataset ID AUC U #> 1 m1 1 0.72 72 #>
# as.data.frame 'aucroc' as.data.frame(uauc1)
#> modnames dsids aucs ustats #> 1 m1 1 0.72 72
## It is 2-3 times faster than mode = "rocprc" # A sample of 100,000 samp1 <- create_sim_samples(1, 50000, 50000) # a function to test mode = "rocprc" func_evalmod_rocprc <- function(samp) { curves <- evalmod(scores = samp$scores, labels = samp$labels) aucs <- auc(curves) } # a function to test mode = "aucroc" func_evalmod_aucroc <- function(samp) { uaucs <- evalmod(scores = samp$scores, labels = samp$labels, mode="aucroc") as.data.frame(uaucs) } # Process time system.time(res1 <- func_evalmod_rocprc(samp1))
#> user system elapsed #> 0.033 0.004 0.037
system.time(res2 <- func_evalmod_aucroc(samp1))
#> user system elapsed #> 0.018 0.001 0.019
# AUCs res1
#> modnames dsids curvetypes aucs #> 1 m1 1 ROC 0.5005588 #> 2 m1 1 PRC 0.4992248
res2
#> modnames dsids aucs ustats #> 1 m1 1 0.5005588 1251396985