Ingestion pipelines and keeping an index in sync

Building a RAG index once is a weekend. Keeping it correct while the sources change underneath it is the job, and it is where most production RAG systems quietly rot: documents get edited, deleted and re-permissioned, and the index keeps answering from a version of the world that no longer exists.

The framing that makes the rest obvious: your index is a cache of a source of truth you do not own. Every cache problem applies — invalidation, staleness, and the fact that nothing tells you when it is wrong.

Change detection

Full re-ingestion is correct and, for a small corpus, the right answer. It stops being right when the embedding bill or the crawl window exceeds your tolerance.

updated_at alone is the trap. It catches edits and silently misses deletions, so deleted documents stay retrievable forever. If the source only offers a timestamp, you need a periodic reconciliation pass that lists source ids and removes index entries with no match.

Content hashing earns its keep because updated_at also lies in the other direction: a bulk migration or a re-save touches every row’s timestamp and you re-embed the entire corpus for nothing. Hash the extracted text, skip when unchanged, and the no-op case costs a read instead of an embedding call.

Deterministic chunk IDs make ingestion idempotent

The single most useful design decision. Derive the id from the document and the chunk’s position or content, not from a random UUID:

def chunk_id(doc_id: str, index: int, text: str) -> str:
    # Stable across re-runs; changes only when the chunk's text changes.
    digest = hashlib.sha256(text.encode()).hexdigest()[:12]
    return f"{doc_id}:{index}:{digest}"

With stable ids, re-ingesting a document is an upsert and running the pipeline twice is harmless. With random ids, every re-run duplicates the corpus, and the symptom is retrieval returning the same passage five times while recall quietly drops.

The update that orphans chunks

The failure people miss. A document is edited and gets shorter: 12 chunks become 8. Upserting the 8 new chunks leaves chunks 9–12 in the index, still retrievable, still citing a version of the document that no longer says that.

async def reindex(doc: Document) -> None:
    chunks = chunk(parse(doc))
    new_ids = {chunk_id(doc.id, i, c.text) for i, c in enumerate(chunks)}

    # Delete-then-upsert scoped to the document, not the whole index.
    existing = await store.list_ids(filter={"doc_id": doc.id})
    await store.delete(ids=set(existing) - new_ids)     # the orphan fix
    await store.upsert(build_records(doc, chunks))

Deleting by doc_id filter before upserting is simpler and usually correct; computing the difference avoids re-writing unchanged chunks, which matters when embeddings are the cost. Either way, scope the delete to the document — a global rebuild inside a per-document handler is an outage waiting for a busy day.

Permissions drift is a security bug, not a freshness bug

If retrieval is filtered by a permission field copied into the index at ingest time, then a document that becomes restricted stays visible until it is re-ingested. That is a data leak with a delay fuse.

Two honest options: re-check permissions at query time against the source of truth (correct, costs latency), or propagate permission changes as their own event type with a much tighter freshness budget than content changes. Deciding this explicitly — rather than discovering it in an audit — is the senior move. Tenant-level isolation is in Design multi-tenant vector search.

Re-embedding migrations

Changing the embedding model invalidates every vector — they are not comparable across models, so this is a full re-embed. The cost and time at volume are in RAG on large data volumes.

Run it as a shadow index rather than in place:

      writes ──┬──> live index   (old model)  ──> serves queries
               └──> shadow index (new model)  ──> evaluated offline
                                    │
                    recall@k passes ▼ cutover, keep old for rollback

Dual-writing during the backfill is what makes the cutover boring: the new index is already current when you switch, and the old one is still there if recall turns out worse. Evaluate before cutting over, on your own golden set, per Building eval sets — a newer model is not automatically better on your corpus.

Budget the backfill. Embedding a large corpus is a rate-limit and cost event, not a script you run in a terminal. Batch, respect the provider’s limits, make it resumable from a checkpoint, and run it at low priority so it does not starve live ingestion.

Freshness is an SLI

Nothing in a RAG system errors when the index is stale, so measure it:

• Ingestion lag — source modification to index availability, at p95. This is the number to put an SLO on.
• Reconciliation drift — documents in the source with no index entry, and index entries with no source. Both should be near zero; either being non-zero means change detection is broken.
• Failed documents — parse failures silently shrink the corpus. Alert on the rate and keep the list, because “the answer isn’t in there” is unfalsifiable without it.
• Chunk count over time — a sudden drop means a parser regression, a sudden jump means a chunking change nobody flagged.

Where it breaks in production

• Random chunk ids, so every re-run duplicates the corpus.
• Deletes never propagate, so the index answers from documents that no longer exist. The most common correctness bug in production RAG.
• Orphaned chunks after a document shortens.
• Permission changes lag content changes, which is a leak.
• One poison document kills the batch and the run aborts halfway. Isolate per-document failures and continue.
• Backfill starves live ingestion because both share one rate limit.
• No reconciliation, so drift accumulates invisibly for months.

Related

• Structured Document Pipelines — Parsing, Tables, OCR, Extraction - the parse step this pipeline wraps.
• Chunking - what a chunk should be.
• Vector Databases - upsert semantics and the cost of a re-embed.
• What are embeddings - why vectors are not comparable across models.
• Building eval sets
  • proving the new index is not worse.
• The transactional outbox
  • emitting change events reliably.