PII, privacy and the EU AI Act

Article.

Verified 2026-08

Verified 2026-08.

A user asks you to delete their data. In a normal service that is a DELETE across a handful of tables. In an LLM system, name every place their data came to rest and say which of those you can actually honour — that question sorts people who have run one of these in production from people who have read about compliance.

Regulatory timelines shift; the dates below were re-checked 2026-08 and are worth re-verifying before you rely on specifics.

The mechanism: an LLM pipeline is a copying machine

A conventional request reads data and returns it. An LLM request copies it — a prompt is a copy, an embedding is a derived copy, a checkpoint is a copy, a cache entry is a copy, and a log line is a copy.

   user text
      |
      +--> provider API      (their retention window)
      +--> your request log  (often verbatim by default)
      +--> trace / span      (a second vendor)
      +--> agent checkpoint  (full message history)
      +--> semantic cache    (keyed by meaning)
      +--> vector index      (derived, still personal)
      +--> fine-tuning set   (no delete path)
      \--> model output ---> wherever that gets written

Each arrow is a new controller obligation and a new place a deletion request has to reach. Most of them were created by a library default rather than by a decision, which is why the honest answer to “where is this user’s data” is usually longer than the architecture diagram suggests.

Which copies can be deleted

The two rows people miss are checkpoints and fine-tuning data. A durable agent persists conversation history to a database, which makes it a personal data store subject to the same rules as any other — and nobody put it on the data map because the framework created it.

Fine-tuning is worse: personal data in the training set is in the weights, and there is no operation that removes it short of retraining from a cleaned corpus. A deletion request against a fine-tuned model is unsatisfiable, which is the actual reason not to fine-tune on personal data — not cost, not quality.

An embedding is not anonymisation

The most common wrong assumption in this area, and worth being able to refute precisely.

“We only store vectors, not text” is offered as a privacy control constantly. It is not one. Embedding inversion research — the vec2text line of work from 2023 onward — reconstructs the source text from its embedding with high fidelity, recovering short passages exactly a large fraction of the time when the attacker knows the embedding model.

# What people think they stored          What it actually is
[0.021, -0.118, 0.334, ...]              "Jane Okafor, NHS no. 4432..."

What follows from that

The consequences are all practical:

• A vector index holding embeddings of personal data is a personal data store. It needs the same access control, retention policy and deletion path as the source table.
• Deletion must reach it. Store a subject reference on every chunk at ingest time, because reconstructing which of four million vectors belong to one person afterwards is not a thing you can do.
• Redact before embedding, not after retrieval. Redaction applied on the way out leaves the identifiers sitting in the index.

class Chunk(BaseModel):
    id: str
    doc_id: str
    subject_ref: str | None   # who this is about; the delete key
    text: str
    embedding: list[float]

async def forget(subject_ref: str) -> None:
    await index.delete(filter={"subject_ref": subject_ref})
    await checkpoints.delete_where(subject_ref=subject_ref)
    await cache.invalidate_by_subject(subject_ref)
    await docs.delete_where(subject_ref=subject_ref)

Writing that function early is the cheap move. Writing it after a regulator asks means discovering which stores have no key to delete by.

Redaction, and how much it buys

# "John Smith called about account 4432" ->
# "[PERSON_1] called about account [ACCOUNT_1]"
redacted, mapping = redact_pii(text)
response = await llm(redacted)
final = restore(response, mapping)

Placeholder-and-restore keeps raw identifiers out of the provider’s context and logs while preserving enough structure for the model to be useful. Detection uses Presidio, spaCy NER or a small classifier.

All of them are recall-limited, so treat this as risk reduction rather than a control — the same distinction as prompt injection. Unusual names, informal references, transliterations and domain-specific identifiers are exactly what NER misses, and a miss is silent.

And redaction does nothing about quasi-identifiers. “The night-shift nurse at the Leeds site who transferred in March” names nobody and identifies one person. Nothing in a PII detector will flag it.

Logging

Do not log prompts and responses verbatim by default. Log token counts, latency, cost, model version and a trace id. Log content only when sampled, redacted, and on a short retention.

The default in most LLM observability tooling is the opposite, and it silently makes your tracing vendor a processor of everything your users typed.

Data residency

If personal data must stay in a region, use a deployment that guarantees it — a region-pinned managed endpoint rather than a public API. This is usually the actual reason enterprises pick a cloud-hosted model over a direct provider API, and it is a better answer than any feature comparison. See Azure for an AWS-shaped brain.

Check the provider’s terms on training from your inputs — business tiers generally exclude it, consumer tiers may not — and on retention. Providers typically keep API inputs for an abuse-monitoring window measured in weeks; zero-retention arrangements exist and generally have to be requested rather than being the default.

The EU AI Act

Risk-tiered regulation with extraterritorial reach: it applies if the system is used in the EU, wherever you are.

What applies right now

The timeline was amended on 16 June 2026, which is the detail separating a current answer from a stale one:

• Most high-risk obligations were delayed. Standalone high-risk systems (Annex III) moved from August 2026 to December 2027; product-embedded high-risk systems (Annex I) moved to August 2028.
• Article 50 transparency duties were NOT delayed and took effect 2 August 2026.
• GPAI enforcement powers and the penalty regime also activated 2 August 2026. The first year was compliance on paper without penalty exposure; that has changed.

So the practical position: transparency obligations are live, penalties are live, and the heavy high-risk conformity work has more runway than originally scheduled.

Article 50, which is most people

1. Disclose that users are interacting with an AI, unless it is obvious.
2. Mark AI-generated content machine-readably.
3. Label deepfakes and synthetic media.

Pre-existing systems have until 2 December 2026 for the machine-readable marking duty specifically. In practice: a visible “AI assistant” label, provenance metadata on generated media, and disclosure in the UI rather than buried in terms.

If you are high-risk

Risk management, data governance, technical documentation, automatic logging of operation, human oversight, accuracy and robustness measures, and conformity assessment before market.

The engineering implication is that audit logging and human oversight stop being nice-to-haves. If the system decides anything about employment, credit or access to essential services, you must be able to show a human could meaningfully intervene, and hold records of what it did.

Other regimes

• GDPR applies independently. Automated decisions with legal or similarly significant effects trigger Article 22 rights, including meaningful information about the logic involved — a real argument for interpretable models in those domains.
• Sector rules — HIPAA, PCI-DSS, financial regulation — apply as they always did. An LLM in the flow exempts nothing.
• Copyright on training data and outputs remains contested. Note it as a risk; do not claim certainty.

Where it breaks in production

The delete that missed three stores

The subject row is gone, the vector index still has forty chunks, the agent checkpoint still has the full transcript, and the semantic cache still returns their data to the next person who asks a similar question.

The deletion path has to be written as one function that fans out, tested, and extended whenever a new store appears. A per-service delete implemented by whoever added that service is how you end up with three of five.

The semantic cache that crosses tenants

A cache keyed on embedding similarity rather than on identity will happily serve tenant A’s cached answer to tenant B, because the questions were similar. It is a correctness bug and a data breach in the same line of code.

key = (tenant_id, user_id, embed(question))   # identity FIRST, then similarity

Scope every cache by tenant, and by user wherever the answer depends on who asked.

Logs in a second vendor

Prompts are redacted before the provider call and the tracing SDK captures the unredacted span attributes anyway. Now the observability vendor holds everything and nobody listed them as a processor.

Redact at the point of capture, not at the point of send, and audit what the tracing integration records by default.

The model repeating what you gave it

Redaction on the way in does nothing if the response is written somewhere durable and the model reconstructed an identifier from context. Anything an LLM writes into a store you keep is data you now hold.

The retention flag applied to one endpoint

Zero-retention is negotiated for the chat endpoint; the embeddings endpoint, the batch API and the evaluation tooling each go through a different path with the default. Enumerate every egress, not every feature.

What to build

The audit trail is the one to mention, because it is simultaneously a compliance answer and ordinary good engineering:

@dataclass(frozen=True)
class DecisionRecord:
    decision_id: str
    subject_ref: str          # pseudonymous, not a name
    model: str                # pinned version, never an alias
    prompt_version: str
    context_ids: list[str]    # what was retrieved, by id
    outcome: str
    reviewed_by: str | None   # the human oversight, as evidence
    at: datetime

Note what it does not contain: the prompt text, the retrieved passages, or the subject’s name. It stores references, so the record itself is not another copy of the personal data.

reviewed_by is what turns “a human could intervene” from a claim into evidence, and pinning model to a version rather than an alias is what lets you answer “what decided this” a year and three provider upgrades later.

The trade not taken

The strongest privacy control is not sending the data at all.

Pseudonymise at the boundary — replace identities with opaque references before anything reaches the pipeline, and resolve them only in the application layer that already had the right to see them. The model works on structure and content, which is usually what it needed; the identifiers never enter a prompt, a log, an index or a cache, and the deletion story collapses back to one table.

The alternative people reach for instead is self-hosting an open-weight model, which removes the provider from the picture. That is a real option and it is not the same as removing the problem: your logs, checkpoints, caches and vector index are all still there, and now you own the infrastructure too. It solves residency and third-party processing; it does nothing about the copying machine.

Deciding which of those you need is the actual design question, and the answer turns on whether the risk is who else holds it or how many places you hold it.

Related

• Prompt injection - the adversarial side of the same boundary
• Model governance and responsible AI - the process around it
• LLM observability - what to log and what not to