Probability and statistics for ML

The part of the maths that shows up in product conversations: is this A/B result real, why is your 99th percentile latency the number that matters, and why does a 99%-accurate fraud model catch almost nothing.

Distributions you should recognise on sight

Latency is log-normal, not normal. That’s why mean latency is a lie and you report p50/p95/p99. A senior answer to “how do you monitor an ML service” mentions percentiles unprompted. See SLO, SLI, SLA, and Error Budgets.

Expectation and variance

E[X]     = sum(x * p(x))          # the mean, the "centre of mass"
Var[X]   = E[(X - E[X])**2]       # spread, in squared units
Std[X]   = sqrt(Var[X])           # spread, in original units

Two properties that get used constantly:

• Linearity of expectation: E[X + Y] = E[X] + E[Y] — always, even when X and Y are dependent. This makes many “expected number of…” puzzles trivial.
• Variance is not linear: Var[X + Y] = Var[X] + Var[Y] only when independent. Correlated failures are why “we have three replicas so we’re 99.9%^3 safe” is wrong.

Bayes’ theorem, and the base rate trap

P(A|B) = P(B|A) * P(A) / P(B)

The classic interview scenario: a fraud detector with 99% sensitivity and 99% specificity, on a population where 0.1% of transactions are fraud.

prior_fraud      = 0.001
p_flag_if_fraud  = 0.99      # sensitivity / recall
p_flag_if_clean  = 0.01      # 1 - specificity

p_flag = p_flag_if_fraud * prior_fraud + p_flag_if_clean * (1 - prior_fraud)
precision = p_flag_if_fraud * prior_fraud / p_flag
# 0.0902 -> only ~9% of flagged transactions are actually fraud

91% of your alerts are false positives, from a model that sounds excellent. This is the single most useful piece of statistics for an ML interview: it explains why accuracy is a useless metric on imbalanced data, why precision/recall exist, and why the fraud team hates your model. Continued in Precision, recall and F1.

Central limit theorem

The mean of many independent samples is approximately normal, regardless of the underlying distribution. This is what licenses confidence intervals and t-tests on non-normal data — you’re not assuming the data is normal, you’re relying on the sample mean being normal.

The caveat that matters: it needs independence and finite variance. Heavy-tailed data (power laws) converges slowly or not at all, which is why “average revenue per user” is unstable when a few whales dominate.

Confidence intervals

import numpy as np

def mean_ci(sample: np.ndarray, z: float = 1.96) -> tuple[float, float]:
    """95% CI for the mean. z=1.96 for 95%, 2.576 for 99%."""
    m = sample.mean()
    se = sample.std(ddof=1) / np.sqrt(len(sample))
    return m - z * se, m + z * se

The standard error shrinks as 1/sqrt(n). To halve your error bar you need four times the data — the reason experiments take longer than product managers expect.

A 95% CI does not mean “95% probability the true value is in this interval”. It means the procedure produces intervals that contain the true value 95% of the time. Being able to state that correctly is a mild seniority signal.

Hypothesis testing and p-values

A p-value is P(data at least this extreme | null hypothesis true). It is not the probability the null is true, and not the probability your result is a fluke.

Practical failure modes worth naming in an interview:

• Peeking. Checking the test daily and stopping when it goes significant inflates false positives badly. Fix: fixed sample size decided in advance, or a sequential-testing method built for continuous monitoring.
• Multiple comparisons. Test 20 metrics at alpha = 0.05 and you expect one false positive by construction. Fix: Bonferroni (conservative) or Benjamini-Hochberg (controls false discovery rate).
• Statistical vs practical significance. With ten million users, a 0.01% lift is significant and worthless. Always quote the effect size and its CI, not just the p-value.

For bootstrapping, which sidesteps most distributional assumptions:

def bootstrap_ci(sample, statistic=np.mean, n_boot=10_000, seed=0):
    rng = np.random.default_rng(seed)
    boots = [statistic(rng.choice(sample, size=len(sample), replace=True))
             for _ in range(n_boot)]
    return np.percentile(boots, [2.5, 97.5])

Bootstrapping is the pragmatic default when you can’t justify a parametric test — it works for medians, percentiles and weird custom metrics where no closed form exists.

Correlation is not causation, and the ways it bites

• Confounding: ice cream sales and drownings correlate; temperature causes both.
• Selection bias: your training data is the users who didn’t churn before you logged them.
• Simpson’s paradox: a trend present in every subgroup reverses when the groups are pooled. It’s the reason segment-level analysis is mandatory before you believe an aggregate.

The practical version for an ML engineer: a feature that correlates with the label in training may be a consequence of it, not a cause. That’s target leakage, and it produces spectacular offline metrics and a useless production model. See Data leakage.