---
name: Model Evaluation
description: Evaluate on data the model never saw.
---
# Model Evaluation

A model's training accuracy tells you it fit the data, not that it generalizes. Evaluation must be on data the model never saw, in a way that mirrors how it'll be used.

## Method
- **Hold out a test set, touched once.** Train on training, tune hyperparameters on validation, report only on test. Touching test repeatedly leaks information.
- **Match the split to the use case.** Random split for iid data; **time-based split** for forecasting (train on past, test on future — never the reverse); grouped split when predictions span entities.
- **Pick the metric that matches the cost.** Accuracy misleads on imbalanced classes; use precision/recall/F1 or AUC. For ranking, NDCG. For regression with outliers, MAE over RMSE.
- **Baselines first.** Beat a sensible baseline (mean, last-value, simple heuristic). A complex model that doesn't beat the baseline is noise with extra compute.