Skip to contents

Compare several models

Source: vignettes/articles/howto-multiple-models.Rmd
howto-multiple-models.Rmd

Several models tested on the same data give one curve each, drawn together on one plot.

library(precrec)
library(ggplot2)

samps <- create_sim_samples(1, 100, 100, "all")

mdat <- mmdata(samps[["scores"]], samps[["labels"]],
  modnames = samps[["modnames"]]
)

The five models here are the five simulated quality levels, from random to perfect.

Calculate and plot

evalmod() notices there is more than one model and labels the curves accordingly.

curves <- evalmod(mdat)

autoplot(curves)

Compare the areas 

knitr::kable(auc(curves))
modnames dsids curvetypes aucs baselines
random 1 ROC 0.4971000 0.5
random 1 PRC 0.4992116 0.5
poor_er 1 ROC 0.8328000 0.5
poor_er 1 PRC 0.7860641 0.5
good_er 1 ROC 0.8180000 0.5
good_er 1 PRC 0.8574152 0.5
excel 1 ROC 0.9780000 0.5
excel 1 PRC 0.9782574 0.5
perf 1 ROC 1.0000000 0.5
perf 1 PRC 1.0000000 0.5

The gap between the two curve types is the point of the package: ROC areas stay high for models the precision-recall areas show to be weak. See balanced and imbalanced data.

One curve type at a time 

autoplot(curves, "PRC")

Naming the models

Without modnames, models are named m1, m2 and so on. Pass your own names to mmdata() or straight to evalmod().

curves2 <- evalmod(
  scores = samps[["scores"]], labels = samps[["labels"]],
  modnames = c("random", "poor", "good", "excellent", "perfect")
)

Next