precrec: A package for computing accurate ROC and Precision-Recall curves

The precrec package contains several functions and S3 generics to provide a robust platform for performance evaluation of binary classifiers.

Functions

The precrec package provides the following six functions.

Function	Description
`evalmod()`	Main function to calculate evaluation metrics
`mmdata()`	Reformat input data for performance evaluation calculation
`join_scores()`	Join scores of multiple models into a list
`join_labels()`	Join observed labels of multiple test datasets into a list
`create_sim_samples()`	Create random samples for simulations
`format_nfold()`	Create n-fold cross validation dataset from data frame

S3 generics

The precrec package provides nine different S3 generics for the S3 objects generated by the evalmod() function.

S3 generic	Library	Description
`print`	base	Print the calculation results and the summary of the test data
`as.data.frame()`	base	Convert a precrec object to a data frame
`plot()`	graphics	Plot performance evaluation metrics
`autoplot()`	ggplot2	Plot performance evaluation metrics with ggplot2
`fortify()`	ggplot2	Prepare a data frame for ggplot2
`auc()`	precrec	Make a data frame with AUC scores
`part()`	precrec	Calculate partial curves and partial AUC scores
`pauc()`	precrec	Make a data frame with pAUC scores
`auc_ci()`	precrec	Calculate confidence intervals of AUC scores

Performance metric calculations

The evalmod() function calculates ROC and Precision-Recall curves and returns an S3 object. The generated S3 object can be used with several different S3 generics, such as print and plot(). The evalmod() function can also calculate basic evaluation metrics - error rate, accuracy, specificity, sensitivity, precision, Matthews correlation coefficient, F-score, balanced accuracy, negative predictive value, informedness, markedness, and Cohen's kappa.

Data preparation

The mmdata() function creates an input dataset for the evalmod() function. The generated dataset contains formatted scores and labels.

join_scores() and join_labels() are helper functions to combine multiple scores and labels.

The create_sim_samples() function creates test datasets with five different performance levels.

Data visualization

plot() takes an S3 object generated by evalmod() as input and plot corresponding curves.

autoplot() uses ggplot to plot curves.

Result retrieval

as.data.frame() takes an S3 object generated by evalmod() as input and and returns a data frame with calculated curve points.

auc() and pauc() returns a data frame with AUC scores and partial AUC scores, respectively. auc_ci() returns confidence intervals of AUCs for both ROC and precision-recall curves.

average_precision() returns the step estimator of the area under the precision-recall curve, and prbe() the precision-recall break-even point.

Probability-based metrics

prob_metrics() calculates the Brier score and the log loss of prediction scores that are probabilities, and prob_metrics_ci() returns their confidence intervals over multiple test datasets.

Uncertainty from one test set

auc_boot() resamples a single test set so that auc_ci() can put a percentile interval around its AUC, and auc_diff() can compare two models on the same resamples.

Classification report

classification_report() reports precision, recall and the F-score of every class at one operating point, in the layout scikit-learn's function of that name prints.

Author

Maintainer: Takaya Saito takaya.saito@outlook.com (ORCID)

Authors:

Takaya Saito takaya.saito@outlook.com (ORCID)
Marc Rehmsmeier marc.rehmsmeier@ii.uib.no (ORCID)