Performance metrics are used to measure how well a machine learning model can perform a task.

It is important to have multiple metrics to evaluate the model’s overall performance because most of the metrics only focus on one aspect of the performance.

The baseline score / baseline model is a simple model that provides a base level of performance to compare against more complex machine learning models. It is used for bench marking the additional value beyond what the simple approach can provide.

Examples:

• Classification: Predicts the most frequent class.
• Regression: Predicts a central tendency (mean, median, mode).

$$ MSE = \frac{1}{n}\sum (y_i-\hat{y_i})^2 $$

• Use MSE when you need to penalise large errors or for optimisation purposes.

$$ RMSE = \sqrt{\frac{1}{n}\sum (y_i-\hat{y_i})^2} $$

• Use RMSE when you want to penalise large errors, but see it in the unit of the target, making it interpretable.

$$ MAE = \frac{1}{n}\sum |y_i-\hat{y_i}| $$

• Use MAE when all errors, large or small, have equal importance.

$$ Max(y_i-\hat{y_i}) $$

• Use Max Error to identify and quantify the largest errors or outliers in your predictions

Image Source: https://365datascience.com/tutorials/statistics-tutorials/sum-squares/

• Sum of Squared Total: $SST = \sum (y_i-\bar{y})^2$ or $SST = SSR +SSE$
  • SST measure the difference between actual dependent value against the mean dependent values
• Sum of Squared Regression: $SSR = \sum (\hat{y_i}-\bar{y})^2$
  • SSR measures the predicted value against the mean dependent values.
• Sum of Squared Error: $SSE = \sum (y_i -\hat{y_i})^2$
  • SSE measures the predicted value against the dependent values.

$$ R^2=\frac{SST-SSE=SSR}{SST} $$

• Definition: R-squared is a measure that represents the proportion of the variance for a dependent variable that’s explained by the independent variable(s) in a regression model.
• Intuition: R-squared provides insight into how well the independent variable(s) in a regression model account for the variability in the dependent variable.
• Application: It used to test goodness of fit and be can used to compare with other model’s performance.
• Example: R-squared of 0.75, implies that 75% of the variation in the dependent variable can be explained by the independent variables.

The higher $R^2$, the better it is because it implies that most of the variation is being explained by the regression line, which also means less prediction error as $SSE$ is lower.

Classification metrics measures the performance when it comes to assigning observations to certain classes.

• True Positive - The prediction is positive and it is correct.
• False Positive - The prediction is positive but it is actually negative.
• True Negative - The prediction is negative and it is correct.
• False Negatives - The prediction is negative but it is actually positive.

Intuition: % of the correction prediction (Correct Prediction/Total Prediction)

$\text{Accuracy Rate} = \frac{TP+TN}{TP+FN+FP+TN}$

$\text{Error Rate (1-Accuracy Rate)} = \frac{FP+FN}{TP+FN+FP+TN}$

It is appropriate to use these metrics when the target classes are well balanced, labels. For example, to classify if it’s cat or dog, need to have equal number of cats or dogs for accuracy or error rate to be useful.

$$ \text{Recall Rate} = \frac{TP}{TP+FN} = \frac{\text{True Positives}}{\text{All Actual Positives}} $$

• Definition: Ratio between true positive and all actual positives.
• Intuition: The ability to correctly identify all the actual positives (true positives and false negative). Recall attempts to answer “What proportion of actual positives was identified correctly?”
• Example: A recall of 11% implies that the model correctly identifies 11% of all the true positives.

$$ \text{Precision Rate} = \frac{TP}{TP+FP} = \frac{\text{True Positives}}{\text{All predicted positives}} $$

• Definition: Ratio between true positive and all predicted positives.
• Intuition: The ability to make correct predictions. Precision attempts to answer “What proportion of predicted positives are actually true positives?”
• Example: A precision of 0.5 implies that the model correctly predicts 50% of the time.

• Recall is a measures the ability of the model to correctly identify all the actual spam emails.
• Precision is a measure of how many of the emails classified as spam by the model are actually spam.

Precision and Recall are inversely related because increasing the threshold increases false negative rate which will worsen recall and concurrently decrease false positive, which improves precision.

When threshold increases:

• false negative rate increases because the model has a higher chance of rejecting (classifying as negative) when it is actually positive.
• false positive rate decreases because the model is more stringent in approving (classifying as positive), making it more accurate in classifying the positive class.

• False Negative: Type 2 Error.
• False Positive: Type 1 Error.
• Threshold: Significance Level.

Similar to increasing the probability threshold, increasing the significance level will lead to less false positive and more false negative.

Prioritise precision over recall when minimising false positive errors is more critical than capturing all positive instances.

Examples:

• Medical Treatment: False positives in this context could lead to unnecessary treatments or distress for patients.
• Legal Proceedings: False positives in this context could lead the incorrectly imprisoning innocent people.

Prioritise recall over precision when it is more critical to capture all or most positive instances.

Examples:

• Medical Screening: Disease screening, such as mammography for breast cancer detection. In this case, missing a true positive (not detecting cancer when it’s present) can be devastating.
• Fraud Detection: It is better to flag more claims as positive (for further investigation) than to not detect it. That said, we should not allow too many false positives as it may harm user’s experience.

If the dataset is unbalanced, the trade-off between precision and recall will be greater. Precision tends to be worse off because you’re more likely to get false positives due to lack of data to sufficiently train the dataset.

Mitigate this issue by:

• changing the threshold conditions
• balancing the dataset.

$$ F_1 = 2 * \frac{P*R}{P+R} $$

• Definition: The F1 Score combines precision and recall into a single value. It provides a balance between these two metrics, allowing you to assess the model’s overall performance.
• Intuition: The ability correctly identify all the positive examples without making too many false positives.
• Example: A F1 score of 0.8 implies that the model is able to capture most of positive instances without making too many false positives.

The harmonic mean was used instead of simple average to reflect the importance of having good scores for both metrics by penalising extremely low values.

For example, when a model exhibits a precision of 1.0 and a recall of 0.0, the simple average of these values would be 0.5. However, the F1 score, which is 0 in this case, is more influenced by the lower of the two values.

$\text{False Positive Rate = }\frac{FP}{FP+TN} = \frac{\text{False Positives}}{\text{All Actual Negatives}}$

• Definition: The ratio of false positives over all actual number negatives.
• Intuition: The False Positive Rate quantifies how often a model makes the mistake of labelling something as positive when it is actually negative.

Image Source: https://towardsdatascience.com/how-to-evaluate-a-classification-machine-learning-model-d81901d491b1

$\text{False Positive Rate = }\frac{FP}{FP+TN} = \frac{\text{False Positives}}{\text{All Negatives}}$

$\text{True Positive Rate (Recall Rate)} = \frac{TP}{TP+FN} = \frac{\text{True Positives}}{\text{All Positives}}$

When the threshold is set to 0, both True Positive Rate (TPR) and False Positive Rate (FPR) will be 1, as there will be no negatives (False Negatives and True Negatives).

As the threshold increases:

• The True Positive Rate (TPR) decreases because the higher rejection rate results in an increase in false negatives.
• The False Positive Rate (FPR) decreases because the higher rejection rate leads to a decrease in false positives.

$J=\text{Sensitivity}+\text{Specificity}−1=TPR+TNR−1$

Optimal Threshold: Select the threshold that yields the highest J statistic value, which corresponds to the greatest gap between true positive rate (TPR) and false positive rate (FPR). A larger gap between TPR and FPR maximises the correct classification of positive cases while minimising false positives.

• AUC = 1: Everything was classified correctly.
• AUC = 0.5: Model has no capacity to distinguish the classes
• AUC = 0: Everything was classified incorrectly.

Image Source: https://machinelearningmastery.com/roc-curves-and-precision-recall-curves-for-imbalanced-classification/

$\text{Precision Rate} = \frac{TP}{TP+FP} = \frac{\text{True Positives}}{\text{All Predicted Positives}}$

$\text{ Recall Rate} = \frac{TP}{TP+FN} = \frac{\text{True Positives}}{\text{All Actual Positives}}$

AUPRC conveys the quality of a binary classification model’s predictions, whether the model can correctly identify all the positive examples without making too many false positives (accidentally marking too many negative examples as positive).

• Recall score = 1 because there are 0 false negatives.
• Precision score = 0 (close to 0) because there are a lot of false positives since everything will be classified as positive.

As the threshold increases:

• recall decreases because there will be more false negatives.
• precision increases because there will be less false positives.

When the data is imbalanced, false positive rate will be low because there is a much higher proportion of negatives than positives. Therefore AUROC score will be biased when the dataset is imbalanced. In contrast, AUPRC uses precision and recall, which presents a more reliable metric to evaluate the model performance as the precision accounts for the accuracy of the predictions.

Use the F1 score to select the right AUPRC threshold. If we are interested in a threshold that results in the best balance of precision and recall, then this is the same as optimising the F-measure.

Hamming Loss.