Model evaluation
Metrics are where modelling meets the business. Most questions here are really “do you understand what your number means to someone who isn’t an ML engineer”.
| # | File | The question it answers |
|---|---|---|
| 01 | Confusion matrix and accuracy | the four cells everything derives from; why accuracy misleads |
| 02 | Precision, recall and F1 | the trade-off, and that it’s a threshold choice not a model property |
| 03 | ROC-AUC vs PR-AUC | the discriminator question — when ROC-AUC flatters a bad model |
| 04 | Regression metrics | MAE vs RMSE, why MAPE is a trap, asymmetric costs |
| 05 | Calibration | when the probability has to be a real probability |
| 06 | Choosing the metric | working backwards from the decision to the metric |
The two ideas that carry most of this
The threshold is not part of the model. Train once, choose the operating point separately from a business constraint — review capacity, a precision floor, or a cost ratio. Leaving it at 0.5 is a decision by default.
ROC-AUC hides false positives under class imbalance. FPR divides by the total actual negatives, so a huge easy-negative pool absorbs enormous numbers of false alarms with barely any movement. PR-AUC has no true-negative term and reflects what the person reviewing the alerts experiences.