AI & ML / Classical ML / 13_classical_nlp.md

Classical NLP

Updated 6 interview angles 5 min read source
On this page7
  1. The pipeline
  2. Bag of words and TF-IDF
  3. Word embeddings, the middle era
  4. The tasks
  5. When classical still wins
  6. Related
  7. Interview angle

Classical NLP

The text toolkit that predates transformers. It matters for two reasons: job descriptions still say “CV, NLP, ASR” meaning the whole discipline, and a surprising share of production text problems are still better served by TF-IDF and logistic regression than by an LLM.

The pipeline

Classical NLP turns text into a fixed-width numeric vector, then hands it to an ordinary classifier.

text 
raw text -> normalise -> tokenise -> vectorise -> model

Normalisation is lowercasing, stripping punctuation, and sometimes collapsing word forms:

  • Stemming chops suffixes with rules. Fast, crude, produces non-words - studies becomes studi.
  • Lemmatisation maps to a dictionary form using part-of-speech context. Slower, correct - studies becomes study, better becomes good.

Use lemmatisation when the output is read by humans or fed to a rule, stemming when it is only ever a feature.

Tokenisation splits into units. Whitespace splitting fails on contractions, hyphenation and any language without spaces, which is why subword tokenisers (BPE, WordPiece) won - they have no out-of-vocabulary problem. See Tokenization.

Bag of words and TF-IDF

The vectorisation step, and the one worth understanding properly.

Bag of words counts occurrences and discards order. "dog bites man" and "man bites dog" produce identical vectors - which is the model’s central limitation, and the reason n-grams exist as a partial fix.

TF-IDF weights each term by how informative it is:

text 
tfidf(t, d) = tf(t, d) * log(N / df(t))

Term frequency rewards words common in this document; inverse document frequency penalises words common across all documents. So “the” appears everywhere, gets a near-zero IDF, and contributes nothing — without a stopword list doing it by hand.

python 
from sklearn.feature_extraction.text import TfidfVectorizer

vec = TfidfVectorizer(ngram_range=(1, 2), min_df=3, max_df=0.8, sublinear_tf=True)
X = vec.fit_transform(train_texts)

Three parameters do most of the work. min_df=3 drops terms appearing in fewer than three documents, which removes typos and kills most of the vocabulary explosion. max_df=0.8 drops terms in over 80% of documents — an automatic, corpus-specific stopword list. sublinear_tf applies 1 + log(tf), so a word appearing fifty times is not fifty times as important as one appearing once.

Fit the vectoriser on training data only. Calling fit_transform on the full dataset before splitting leaks test-set vocabulary and IDF statistics into training, and inflates your validation score. It is the most common leak in text pipelines because it does not look like leakage.

Word embeddings, the middle era

Word2Vec and GloVe map each word to a dense vector where distance means similarity. They fixed bag-of-words’ blindness to synonyms, and they have one fatal limitation: one vector per word, regardless of context. “Bank” in river bank and savings bank gets the same vector.

That is precisely what contextual embeddings from transformers solved, and it is the cleanest one-line answer to “why did transformers displace Word2Vec”.

The tasks

Task Classical approach
Classification TF-IDF plus linear model
Named entity recognition CRF over token features
Topic discovery LDA or NMF
Similarity Cosine over TF-IDF or embeddings

Named entity recognition — finding people, organisations and dates — was conditional random fields over hand-built features. A CRF models the whole tag sequence rather than each token independently, which is what stops it emitting an I-PER tag that does not follow a B-PER.

When classical still wins

This is the part that reads as senior rather than dated.

  • Latency and cost. TF-IDF plus logistic regression classifies in microseconds on a CPU, for nothing. An LLM call is 100ms and a bill. At millions of documents that difference decides the architecture.
  • Explainability. You can show the coefficients: this was flagged because of these words. Regulated domains often require that, and “the model said so” does not qualify.
  • Small labelled datasets with a fixed label set. With a few thousand labelled examples and stable classes, a linear model over TF-IDF is frequently competitive with a fine-tune and takes an afternoon.
  • Determinism. The same input gives the same output forever, with no provider, no version drift, no temperature.

The honest framing: use an LLM when the task needs world knowledge, generation, or generalising to labels you did not train on. Use a classifier when the task is “sort these into five buckets you have examples of”. Reaching for an LLM to do sentiment on a fixed taxonomy is the modern equivalent of reaching for a neural net in 2015 — it works, and it is the expensive answer to a solved problem.

The strong hybrid: use an LLM to label a few thousand examples, then train a small classical model on those labels and serve that. You get the LLM’s quality at the classifier’s cost and latency.

Interview angle 6

  • “What is TF-IDF?” - term frequency times inverse document frequency: a word is weighted by how often it appears in this document and how rare it is across the corpus. Words appearing everywhere get near-zero weight, which is stopword removal falling out of the maths.
  • “Stemming or lemmatisation?” - stemming chops suffixes by rule and is fast but produces non-words; lemmatisation maps to a dictionary form using part of speech and is slower but correct. Lemmatise when a human or a rule reads the output.
  • “Why did embeddings replace bag of words, and transformers replace embeddings?” - bag of words discards order and cannot see synonyms; Word2Vec gave dense vectors where distance means similarity but assigns one vector per word regardless of context, so river bank and savings bank collide. Contextual embeddings fixed exactly that.
  • “What is the classic leak in a text pipeline?” - fitting the vectoriser before splitting. fit_transform on the whole dataset leaks test vocabulary and IDF statistics into training and inflates validation scores, and it does not look like leakage.
  • “When would you not use an LLM for text classification?” - fixed label set with labelled examples, high volume, tight latency or cost budgets, or a requirement to explain the decision. TF-IDF plus a linear model runs in microseconds on CPU and its coefficients are the explanation.
  • “Is there a middle path?” - label a few thousand examples with an LLM, train a small classical model on those labels, serve the small model. LLM-quality labels at classifier cost and latency.
Previous
Time series forecasting