tidyposterior

Introduction

This package can be used to conduct post hoc analyses of resampling results generated by models.

For example, if two models are evaluated with the root mean squared error (RMSE) using 10-fold cross-validation, there are 10 paired statistics. These can be used to make comparisons between models without involving a test set.

There is a rich literature on the analysis of model resampling results such as McLachlan's Discriminant Analysis and Statistical Pattern Recognition and the references therein. This package follows the spirit of Benavoli et al (2017).

tidyposterior uses Bayesian generalized linear models for this purpose and can be considered an upgraded version of the caret::resamples() function. The package works with rsample objects natively but any results in a data frame can be used.

Installation

You can install the released version of tidyposterior from CRAN with:

install.packages("tidyposterior")

Install the development version from GitHub with:

# install.packages("devtools")
devtools::install_github("tidymodels/tidyposterior")

Example

library(tidyposterior)
# See ? precise_example
data(precise_example)

# Get classification accuracy results for analysis

library(dplyr)
accuracy <- precise_example %>%
   select(id, contains("Accuracy")) %>%
   setNames(tolower(gsub("_Accuracy$", "", names(.)))) 
accuracy

# Model the accuracy results
acc_model <- perf_mod(accuracy, seed = 13311, verbose = FALSE)   

# Extract posterior distributions:
accuracy_dists <- tidy(acc_model)

# Credible intervals for accuracy per model
summary(accuracy_dists)

Contributing

This project is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.

• For questions and discussions about tidymodels packages, modeling, and machine learning, please post on RStudio Community.

• If you think you have encountered a bug, please submit an issue.

• Either way, learn how to create and share a reprex (a minimal, reproducible example), to clearly communicate about your code.

• Check out further details on contributing guidelines for tidymodels packages and how to get help.