AI & ML / ML foundations / 07_hyperparameter_tuning.md

Hyperparameter tuning

Updated 6 interview angles 5 min read source
On this page7
  1. The search strategies
  2. Optuna in practice
  3. Where the score comes from is the real question
  4. What to tune first
  5. When not to tune
  6. Related
  7. Interview angle

Hyperparameter tuning

Parameters are learned from data; hyperparameters are chosen by you — tree depth, learning rate, regularisation strength, k. Tuning is the search over those choices, and the interesting part is not the search algorithm but where the score comes from.

The search strategies

Strategy How it picks Cost
Grid every combination explodes
Random sampled independently fixed budget
Bayesian / TPE models past trials fewer trials
Hyperband / ASHA kills bad runs early best per GPU-hour

Why random beats grid

The counterintuitive result, and a good thing to be able to explain: with the same budget, random search usually wins.

text 
grid, 9 trials          random, 9 trials
lr:  3 distinct values  lr:  9 distinct values
reg: 3 distinct values  reg: 9 distinct values

A grid of 3x3 spends nine runs learning only three values of each hyperparameter. Random sampling spends the same nine runs learning nine values of each. Since typically only one or two hyperparameters actually matter, the grid wastes most of its budget re-testing the irrelevant one at values you already tried.

Grid search survives for two or three genuinely discrete options. Past that it is the wrong tool.

Optuna in practice

The default answer for Python. Its distinguishing feature is define-by-run: the search space is expressed in ordinary code, so it can branch.

python 
import optuna

def objective(trial):
    depth = trial.suggest_int("max_depth", 3, 12)
    lr = trial.suggest_float("lr", 1e-3, 0.3, log=True)
    model = XGBClassifier(
        max_depth=depth, learning_rate=lr
    )
    return cross_val_score(model, X, y, cv=5).mean()

study = optuna.create_study(direction="maximize")
study.optimize(objective, n_trials=100)
study.best_params

Three details that matter:

  • log=True on a learning rate. Sampling uniformly in [1e-3, 0.3] puts almost every sample above 0.05. Learning rates, regularisation strengths and anything spanning orders of magnitude want log scale.
  • Pruning. study.optimize(..., pruner=MedianPruner()) plus trial.report() stops a trial that is already losing, which is where most of the wall-clock saving comes from.
  • The default sampler is TPE, not random, so the last trials genuinely exploit what the first ones learned.

Note: as of 2026-08, Optuna is 4.9 and supports Python 3.9-3.14. scikit-learn’s GridSearchCV and RandomizedSearchCV are still fine for a small sweep with no extra dependency; HalvingRandomSearchCV gives you successive halving in the stdlib-adjacent option.

Where the score comes from is the real question

Every hyperparameter choice made against a set consumes some of that set’s independence. Tune on validation, report on test, and never iterate against test — see Train / validation / test splits.

The leakage trap

python 
# WRONG — scaler saw the validation fold
X = StandardScaler().fit_transform(X)
cross_val_score(model, X, y, cv=5)

# RIGHT — fitted inside each fold
pipe = make_pipeline(StandardScaler(), model)
cross_val_score(pipe, X, y, cv=5)

Any fitted preprocessing — scaling, imputation, feature selection, target encoding — must live inside the pipeline so it is refitted per fold. Tuning amplifies this: you are now selecting the model that best exploits the leak. See Data leakage.

Nested CV, and when it is worth it

The honest way to estimate performance of the whole tuning procedure is an outer CV loop around the tuning loop. It costs outer x inner fits, which is why it is reserved for small data and for papers rather than for a production sweep with a held-out test set.

What to tune first

Budget is finite, so order matters more than coverage.

Model Tune first
Gradient boosting learning rate, then depth
Random forest depth, min_samples_leaf
Linear + regularised the penalty strength
Neural net learning rate, then batch size

For boosting, learning rate and number of trees trade against each other: halve the rate and you need roughly twice the trees, so tune the rate with early stopping choosing the count rather than gridding both.

When not to tune

Tuning has the worst effort-to-gain ratio of anything in the modelling pipeline, and it is where people hide from harder work:

  1. Fix the data first. Leakage, label noise and a bad split move the number far more than any hyperparameter.
  2. Better features beat better hyperparameters, consistently, on structured data.
  3. A different model class is usually a bigger jump than tuning the current one — try gradient boosting before perfecting the random forest.
  4. Watch for tuning to noise. If your validation set is 200 rows, a 1% improvement across 300 trials is selection on noise, and it will not survive contact with the test set.

Interview angle 6

  • “Grid search or random search?” - random, at equal budget. A 3x3 grid spends nine runs learning three values of each hyperparameter; nine random draws learn nine values of each. Since usually only one or two hyperparameters matter, the grid wastes most of its budget. Grid is fine for two or three genuinely discrete options.
  • “How does Bayesian optimisation improve on that?” - it models the score as a function of the hyperparameters and samples where the expected improvement is highest, so later trials exploit what earlier ones learned. Optuna’s default TPE sampler does this; the win is fewer trials for the same result.
  • “Where does the score for each trial come from?” - cross-validation on the training data, never the test set. Every choice made against a set consumes its independence, which is why the test set is touched once, at the end.
  • “What’s the classic leakage mistake when tuning?” - fitting a scaler, imputer or feature selector on all the data before cross-validating. It must go inside the pipeline so it refits per fold — otherwise you are selecting the hyperparameters that best exploit information from the validation fold.
  • “When would you not bother tuning?” - almost always before fixing the data and the features. Leakage, label noise and a weak feature set dominate hyperparameters, and a different model class is usually a bigger jump than tuning the current one.
  • “How do you know you haven’t tuned to noise?” - the gap between validation and test. Hundreds of trials against a small validation set will find a configuration that fits its noise; a held-out test set scored once is the check, and a large validation-to-test drop is the symptom.
Previous
When not to use ML