Evaluate Your Model with MLmetrics

This post will explore using R’s MLmetrics to evaluate machine learning models. MLmetrics provides several functions to calculate common metrics for ML models, including AUC, precision, recall, accuracy, etc.

Building an example model

Firstly, we need to build a model to use as an example. For this post, we’ll be using a dataset on pulsar stars from Kaggle. Let’s save the file as “pulsar_stars.csv”. Each record in the file represents a pulsar star candidate. The goal will be to predict if a record is a pulsar star based upon the attributes available.

To get started, let’s load the packages we’ll need and read in our dataset.

library(MLmetrics)
library(dplyr)

stars = read.csv(“pulsar_stars.csv”)

Next, let’s split our data into train vs. test. We’ll do a standard 70/30 split here.

set.seed(0)
train_indexes = sample(1:nrow(stars), .7 * nrow(stars))

train_set <- stars[train_indexes,]
test_set <- stars[-train_indexes,]

Now, let’s build a simple logistic regression model.

train_set <- data.frame(train_set %>% select(target_class), train_set %>% select(-target_class))

# build model
model <- glm(formula(train_set), train_set, family = "binomial")

# get predictions on train and test datasets
train_pred <- predict(model, train_set, type = "response")
test_pred <- predict(model, test_set, type = "response")

AUC / precision / recall / accuracy

Let’s calculate a few metrics. One of the most common metrics for classification is calculating AUC, which can be done using MLMetrics’ AUC function. Intuitively, AUC is a score between 0 and 1 that measures how well a model rank-orders predictions. See here for a more detailed explanation.

# get AUC on test and train set
AUC(test_pred, test_set$target_class) # 0.974172
AUC(train_pred, train_set$target_class) # 0.9773794

As a refresher, here’s a quick overview of precision, recall, and accuracy:

• Precision: The true positive rate. If the model predicts there are 10 pulsar stars, and 8 of those 10 actually are pulsars, then the precision would be 8 / 10, or 80%.
• Recall:The proportion of the positive labels that are captured with the model. For example, suppose there are 10 pulsar stars in the data and that the model predicts 7 of those to be pulsar stars. That would mean the recall is 7 / 10, or 70%.
• Accuracy:Generally the most intuitive of the metrics here. Accuracy is simply the number of correct predictions divided by the total number of predictions.

Visit TheAutomatic.net to read the full article:
http://theautomatic.net/2020/01/29/evaluate-your-r-model-with-mlmetrics/