prcbench

R-CMD-check codecov.io CodeFactor CRAN_Status_Badge CRAN_Logs_Badge

The aim of the prcbench package is to provide a testing workbench for evaluating precision-recall curves under various conditions. It contains integrated interfaces for the following seven tools. It also contains predefined test data sets.

Tool Language Link
precrec R Tool web site, CRAN
ROCR R Tool web site, CRAN
PRROC R CRAN
AUCCalculator Java Tool web site
PerfMeas R CRAN
yardstick R Tool web site, CRAN
sklearn Python Tool web site

The sklearn tool uses a standalone Python module bundled with prcbench and derived from the scikit-learn source, so scikit-learn itself is not required. It does need the reticulate package, Python and numpy. Without them it returns a flat dummy curve instead of raising an error, so the predefined tool sets that contain it stay usable.

Timings of sklearn are not comparable with those of the R tools. Every call crosses the R/Python boundary and converts the input and output vectors, and that overhead is counted as part of the measurement. It often dominates the curve calculation itself on small test sets. Use run_benchmark to compare the R tools with each other, and read the sklearn row as the cost of calling Python from R rather than as the speed of the scikit-learn algorithm. Curve accuracy from run_evalcurve is unaffected.

Disclaimer: prcbench was originally develop to help our precrec library in order to provide fast and accurate calculations of precision-recall curves with extra functionality.

Accuracy evaluation of precision-recall curves

prcbench uses pre-defined test sets to help evaluate the accuracy of precision-recall curves.

  1. create_toolset: creates objects of different tools for testing (7 different tools)
  2. create_testset: selects pre-defined data sets (c1, c2, and c3)
  3. run_evalcurve: evaluates the selected tools on the simulation data
  4. autoplot: shows the results with ggplot2 and patchwork 
## Load library
library(prcbench)

## Plot base points and the result of 6 tools on pre-defined test sets (c1, c2, and c3)
toolset <- create_toolset(c(
  "precrec", "ROCR", "AUCCalculator", "PerfMeas", "PRROC", "yardstick"
))
testset <- create_testset("curve", c("c1", "c2", "c3"))
scores1 <- run_evalcurve(testset, toolset)
autoplot(scores1, ncol = 4, nrow = 2)

Running-time evaluation of precision-recall curves

prcbench helps create simulation data to measure computational times of creating precision-recall curves.

  1. create_toolset: creates objects of different tools for testing
  2. create_testset: creates simulation data
  3. run_benchmark: evaluates the selected tools on the simulation data
## Load library
library(prcbench)

## Run benchmark for auc7 (7 tools) on b10 (balanced 5 positives and 5 negatives)
toolset <- create_toolset(set_names = "auc7")
testset <- create_testset("bench", "b10")
res <- run_benchmark(testset, toolset)

print(res)
testset toolset toolname min lq mean median uq max neval
b10 auc7 AUCCalculator 1.11 1.13 1.41 1.23 1.23 2.33 5
b10 auc7 PerfMeas 0.08 0.08 0.11 0.09 0.11 0.21 5
b10 auc7 precrec 6.35 6.43 6.50 6.43 6.44 6.88 5
b10 auc7 PRROC 0.17 0.17 0.20 0.17 0.18 0.30 5
b10 auc7 ROCR 1.81 1.83 1.98 1.88 2.14 2.24 5
b10 auc7 sklearn 0.44 0.46 1.85 0.48 0.52 7.36 5
b10 auc7 yardstick 1.78 1.78 1.94 1.84 1.91 2.36 5

The sklearn row of the table includes the R/Python conversion overhead, so it measures the round trip rather than the scikit-learn algorithm. See the note above.

Documentation

Installation

CRAN 

install.packages("prcbench")

Dependencies

AUCCalculator requires a Java runtime environment (>= 6) if AUCCalculator needs to be evaluated.

GitHub

You can install a development version of prcbench from our GitHub repository.

devtools::install_github("evalclass/prcbench")

  1. Make sure you have a working development environment.

    • Windows: Install Rtools (available on the CRAN website).

    • Mac: Install Xcode from the Mac App Store.

    • Linux: Install a compiler and various development libraries (details vary across different flavors of Linux).

  2. Install devtools from CRAN with install.packages("devtools").

  3. Install prcbench from the GitHub repository with devtools::install_github("evalclass/prcbench").

Troubleshooting

microbenchmark

microbenchmark does not work on some OSs. prcbench uses system.time when microbenchmark is not available.

rJava

  • Some OSs require en extra configuration step after rJava installation.
sudo R CMD javareconf
  • JDKs
  1. Oracle JDK
  2. OpenJDK
  • JDKs for macOS
  1. AdoptOpenJDK
  2. AdoptOpenJDK with homebrew

Citation

Precrec: fast and accurate precision-recall and ROC curve calculations in R

Takaya Saito; Marc Rehmsmeier

Bioinformatics 2017; 33 (1): 145-147.

doi: 10.1093/bioinformatics/btw570