Natural language understanding
The CS224U territory, and the reason it is still worth knowing in 2026: the tasks survived even though the methods collapsed into one. NLI, coreference, semantic parsing and the rest are no longer separate model architectures — they are prompts, or eval sets. Knowing them is knowing how to decompose a vague language requirement into something measurable.
The tasks, and what each actually tests
| Task | Asks |
|---|---|
| NLI / entailment | does A imply B, contradict it, or neither |
| Semantic similarity | how close in meaning are two texts |
| Coreference | which mentions refer to the same entity |
| Semantic parsing | turn language into a formal query or program |
| Relation extraction | which entities are related, and how |
| Sentiment / stance | attitude toward a target |
| QA | answer from a passage, or from nothing |
NLI is the one to know cold, because it is the primitive behind modern practice. “Is this claim supported by this retrieved chunk” is entailment, and that is the faithfulness metric in Why evaluating LLM systems is hard.
# Grounding check, as entailment. Cheap enough to run per claim.
result = nli(premise=retrieved_chunk, hypothesis=generated_claim)
if result.label != "entailment":
# unsupported by the context
flag(generated_claim)A small dedicated NLI model does this at a fraction of an LLM judge’s cost and latency, which matters when you are scoring every sentence of every answer.
Semantic parsing is what a tool call is
Turning “how many orders shipped late last quarter” into an executable query is semantic parsing, and it is exactly what function calling does:
# The schema IS the target formal language.
{"name": "count_orders", "arguments": {"status": "late", "period": "2026-Q1"}}Framing it that way is useful in an interview because it brings the field’s hard-won lessons with it: executable accuracy is the metric, not string match. Two different queries returning the same answer are both correct, and a query that matches the reference string but errors is not. See Function Calling, Tool Use, and Structured Output.
What the benchmarks taught, and why they broke
GLUE and SuperGLUE drove a decade of progress and then saturated — models passed human baselines while still failing obviously. The diagnosis is worth carrying:
- Annotation artefacts. Models learned that a hypothesis containing “not” usually means contradiction, without reading the premise. The dataset had a shortcut and the model found it.
- Distribution narrowness. High scores on the test set, poor behaviour on anything phrased differently.
- Contamination. Benchmarks leaked into pretraining data.
The lesson generalises directly to your own evals: a metric a model can pass without doing the task is not measuring the task. Adversarial and contrast sets — minimally edited examples that flip the label — are the technique that exposes it, and building a few by hand for your own eval set is cheap and revealing.
Where it lands today
| Then | Now |
|---|---|
| A model per task | one model, prompted |
| Task-specific architectures | task-specific evals |
| Feature engineering | prompt and context engineering |
| Benchmark leaderboards | your own domain set |
The practical inheritance is decomposition. When a stakeholder says “understand the customer email”, the useful reply names tasks — intent classification, entity extraction, sentiment toward a target, entailment against policy — because each is separately measurable and separately fixable. That is the whole value of the vocabulary, and it is the same reframing move as Types of ML problem.
Gotcha: “understanding” is not a measurable property. Every claim about it has to reduce to a task with a metric, or it cannot be tested, improved or defended in a review.
Related
Interview angle 5
- “What is NLI and why does it still matter?” - deciding whether a premise entails, contradicts or is neutral toward a hypothesis. It is the primitive behind grounding checks: “is this claim supported by this chunk” is entailment, and a small NLI model runs it far cheaper than an LLM judge.
- “What is semantic parsing?” - turning language into a formal, executable representation. Function calling is semantic parsing, which is why the field’s metric applies: judge on executable accuracy, not string match — two different queries returning the same answer are both right.
- “Why did GLUE and SuperGLUE stop being useful?” - saturation plus annotation artefacts. Models learned shortcuts, like a negation in the hypothesis predicting contradiction without reading the premise, and benchmark data leaked into pretraining. A metric a model can pass without doing the task is not measuring the task.
- “How do you catch a model exploiting a shortcut?” - contrast sets: minimally edited examples that flip the label. If a small edit that changes the answer does not change the prediction, the model is keying on something other than meaning.
- “A stakeholder asks you to ‘understand’ customer emails. What do you do?” - decompose it into measurable tasks: intent classification, entity extraction, sentiment toward a target, entailment against policy. “Understanding” has no metric; each of those does.