- name
SKILL.md
Beceri dosyası yayıncının deposunda kalır.
Kopyalanan komutu bir terminale yapıştırın. Bu SKILL.md dosyasını ilgili ajana kurar.
npx skills add agentsope/SkillAlchemy --skill agentsop-idempotent-ingestion -a cursor -yKopyalanan komutu bir terminale yapıştırın. Bu SKILL.md dosyasını ilgili ajana kurar.
npx skills add agentsope/SkillAlchemy --skill agentsop-idempotent-ingestion -a claude-code -yKopyalanan komutu bir terminale yapıştırın. Bu SKILL.md dosyasını ilgili ajana kurar.
npx skills add agentsope/SkillAlchemy --skill agentsop-idempotent-ingestion -a codex -yKopyalanan komutu bir terminale yapıştırın. Bu SKILL.md dosyasını ilgili ajana kurar.
npx skills add agentsope/SkillAlchemy --skill agentsop-idempotent-ingestion -a grok -yKopyalanan mesajı bir Grok Bot sohbetinde gönderin. Bot bu SKILL.md dosyasını GitHub'dan alır ve tüm Bot'larınızın paylaştığı becerilere ekler.
Install this skill: https://github.com/agentsope/SkillAlchemy/tree/HEAD/skills/agentsop-idempotent-ingestionYayıncıdan
Third-person operating model for a coder agent that owns ingestion correctness across repeated runs. The audience is the LLM agent writing or reviewing the pipeline code — not the end user.
One sentence: The interesting run is the second one. A correct pipeline hashes each document, looks the hash up in a docstore, and inserts / updates / skips accordingly — so re-running over unchanged docs changes nothing.
This skill is an ENHANCE overlay over [[llamaindex]]. The base skill names
IngestionPipeline with docstore + UPSERTS_AND_DELETE as a hardening
must-have (OP-08, anti-pattern A10) but stops at "use it". This overlay
surfaces the correctness contract — what idempotency actually means, how the
hash decides the action, and how it generalises beyond LlamaIndex.
Activate this skill whenever any of the following holds:
VectorStoreIndex.from_documents,
pipeline.run(documents=...), vectorstore.add_documents, a custom embed
RecordManager, "keep the index in sync with the source".Do not activate when:
Three principles. Violating any one means a second run can duplicate or corrupt the index — regardless of how good the retrieval stack downstream is.
Re-running the pipeline over an unchanged document is a no-op. Run it once, run it a thousand times — same index. The mechanism: each document is reduced to a stable content hash (and a stable doc id); that hash is looked up in a docstore (a key→hash record store separate from the vector store); the lookup result decides the action.
doc_id seen? hash changed?
│ │
┌────────────┴───────────┐ ┌──────┴──────┐
no yes no yes
│ │ │ │
INSERT (embed + upsert) │ SKIP UPDATE
└──────────────►(delete old chunks,
re-embed, upsert)
The hash is the whole game. Without it, the pipeline cannot tell "I have seen
this exact content" from "this is new" — so it re-embeds and re-adds
everything, and the vector store accumulates duplicates. (LlamaIndex base skill
failure #3 / A10; developers.llamaindex.ai/.../loading/ingestion_pipeline/.)
The vector store holds embeddings keyed by node id. The docstore holds, per
document, {doc_id → content_hash} (and the node ids derived from it). They are
two stores with two jobs:
The dedup decision happens against the docstore, before any embedding
call — so an unchanged corpus costs zero embedding tokens on re-run. If the
docstore is ephemeral (in-memory, lost on restart), every cold start looks like
a first run and re-embeds the world. The docstore must be persisted to the
same durability tier as the vector store (SimpleDocumentStore.persist(),
Redis, MongoDB, Postgres). A docstore that doesn't survive the process is not a
docstore.
Insert and update are driven by documents that are present. A document that
disappeared from the source emits no event — so a naive upsert pipeline
never learns it should remove the orphaned chunks. Stale content keeps getting
retrieved long after the source file is gone. Propagating deletes requires the
pipeline to compare the full set of doc_ids seen this run against the set in
the docstore, and purge the difference. In LlamaIndex this is the _AND_DELETE
half of DocstoreStrategy.UPSERTS_AND_DELETE; in LangChain it is
cleanup="full" / cleanup="incremental". Choosing the upsert-only strategy
silently accepts stale ghosts — sometimes correct (append-only corpus), usually
not.
Five stages. Each gates the next. Stop and remediate at the first failure. The gate that matters most is Stage 4 — prove the second run is a no-op.
Before code, answer:
doc_id? It must survive across runs —
a file path, a CMS id, a URL. Not a random uuid generated at load time
(that makes every run look new). LlamaIndex derives a hash from content +
doc_id; if doc_id is unstable, idempotency breaks even with a docstore.from llama_index.core.ingestion import IngestionPipeline, DocstoreStrategy
from llama_index.core.storage.docstore import SimpleDocumentStore
from llama_index.core.node_parser import SentenceSplitter
from llama_index.embeddings.openai import OpenAIEmbedding
pipeline = IngestionPipeline(
transformations=[
SentenceSplitter(chunk_size=1024, chunk_overlap=20),
OpenAIEmbedding(model="text-embedding-3-small"),
],
docstore=SimpleDocumentStore(), # the dedup ledger
vector_store=vector_store, # the embedding store
docstore_strategy=DocstoreStrategy.UPSERTS_AND_DELETE,
)
For production, swap SimpleDocumentStore for a server-backed one
(RedisDocumentStore, MongoDocumentStore, PostgresDocumentStore) so the
ledger is shared across workers and survives restarts. A SimpleDocumentStore
that is never .persist()-ed is the #1 cause of "it re-embeds everything every
night".
docs = SimpleDirectoryReader("./data", filename_as_id=True).load_data()
# filename_as_id=True → doc.id_ is the file path → stable across runs.
# LlamaIndex then computes the content hash internally per node.
pipeline.run(documents=docs)
Hard rules:
doc_id comes from a stable external identity (path / CMS id / URL),
never a fresh uuid per load. Set filename_as_id=True or assign doc.id_
explicitly.datetime.now() into metadata) — that makes
the hash change every run and defeats the skip path. Keep volatile metadata
out of the hashed payload.Default for any system that mirrors a mutable source: UPSERTS_AND_DELETE.
DUPLICATES_ONLY is not a re-ingest strategy — it only de-dupes within a
single .run(); a second .run() with the same docs will not skip them.
Before merging, the PR must include — and CI must run — a test that runs the pipeline twice and asserts the second run did nothing observable:
def test_reingest_is_idempotent(pipeline, docs, vector_store):
n1 = len(pipeline.run(documents=docs)) # cold run
count_after_first = vector_store.count()
n2 = len(pipeline.run(documents=docs)) # identical re-run
count_after_second = vector_store.count()
assert n2 == 0, "re-run re-processed unchanged docs"
assert count_after_second == count_after_first, "re-run duplicated vectors"
pipeline.run() returns the list of nodes it actually processed; on a
clean idempotent re-run over unchanged input that list is empty. If it
isn't, the docstore is missing, ephemeral, or the doc ids / hashes are
unstable — go back to Stage 1-2. Then add the edit case (change one doc → exactly
that doc's chunks replaced) and, for UPSERTS_AND_DELETE, the delete case
(remove one doc → its chunks gone, others untouched). See OP-06 / OP-07.
storage_context.persist() /
server-backed store auto-persists) — otherwise the next cold start re-embeds
everything.{docs_in, inserted, updated, skipped, deleted, embedding_calls}. On a steady-state corpus, inserted+updated+deleted should
trend to ~0 and skipped ≈ docs_in. A run that re-embeds everything every
night is the loud signal that idempotency is broken.Format: Trigger / Action / Output / Evidence. Machine-readable in
intermediate/operation_candidates.json.
IngestionPipeline / ingestion path that will run more than
once with no docstore wired.docstore= (persisted: Redis/Mongo/Postgres in prod,
SimpleDocumentStore only for tests) plus docstore_strategy=. The docstore
is the dedup ledger; without it dedup is structurally impossible.developers.llamaindex.ai/python/framework/module_guides/loading/ingestion_pipeline/; [[llamaindex]] OP-08 / A10.filename_as_id=True or assign doc.id_ from a stable
external identity (path / CMS id / URL). Never a fresh uuid per load.Document.doc_id / SimpleDirectoryReader(filename_as_id=True) docs.UPSERTS (append-mostly), UPSERTS_AND_DELETE (mirror of a
shrinkable source — the production default), DUPLICATES_ONLY (in-run dedup
only, not cross-run). Decide from Stage 0 change shape, not by default.DocstoreStrategy enum; LlamaIndex ingestion docs.UPSERTS_AND_DELETE; the pipeline diffs the doc_ids seen this
run against the docstore and purges orphans. (LangChain: cleanup="full" for
a full snapshot run, cleanup="incremental" for streamed batches.)DocstoreStrategy.UPSERTS_AND_DELETE; LangChain index(..., cleanup=...) docs.run() returns 0 processed nodes and the vector count is unchanged (Stage 4).pipeline.run() returns processed nodes; direct application of
Principle 1.UPSERTS_AND_DELETE) → assert its chunks gone, neighbours intact.UPSERTS_AND_DELETE.SimpleDocumentStore used, or docstore not persisted after run.storage_context.persist(persist_dir=...) after each run, or use
a server-backed docstore that auto-persists and is shared across workers.{doc_id → sha256(content)} table next to the vector
store; on each doc compute the hash, compare, and INSERT / UPDATE
(delete-then-add chunks) / SKIP; track seen ids for delete propagation.IngestionPipeline / RecordManager automate; see
R2 cross-framework table.困境: A 200-page handbook had one paragraph edited. The naive idempotent pipeline hashes at the document level → the whole doc's hash changes → all its chunks are deleted and re-embedded. Correct for safety, but on a large doc a one-line edit triggers a full re-embed of that document. Should the pipeline diff at the chunk level instead?
约束:
SentenceSplitter re-segments the whole doc when text shifts, so chunk
boundaries (and ids) move even for unedited text downstream of the edit.决策步骤:
doc_id.结果: Move the granularity by splitting documents, not by hand-rolling chunk-diffing. Document-level hashing stays the default; the fix is upstream in how the source is segmented into docs.
可提取的操作: OP-02, OP-03, OP-04.
困境: 50 files were removed from the source folder. The upsert-only pipeline
leaves their chunks in the vector store forever — they keep surfacing in
retrieval and the LLM cites documents that no longer exist. Switch to
UPSERTS_AND_DELETE? But what if a run sees a partial corpus (a flaky reader
returned only half the files) — would it then delete everything legitimately
missing-this-run?
约束:
UPSERTS_AND_DELETE purges any doc_id in the docstore that was not seen
in the current run. If the run's input is incomplete, that purge is
catastrophic — it deletes live content.cleanup="full"); for streamed
/ partial batches you need cleanup="incremental" (per-batch, scoped by
source).决策步骤:
UPSERTS_AND_DELETE (LlamaIndex) / cleanup="full"
(LangChain). Streamed batches scoped by source → cleanup="incremental".deleted=true metadata +
query-time filter) over hard delete, so erasure is auditable and reversible.结果: Purge — but only behind a "this is a complete snapshot" guard.
Idempotency for deletes is UPSERTS_AND_DELETE plus a partial-input circuit
breaker, never a blind diff-and-delete.
可提取的操作: OP-05, OP-07, OP-08.
UPSERTS (or even
plain add) is sufficient; delete propagation is dead weight.from_documents(...) inside anything that runs on a schedule.IngestionPipeline(...) with no docstore= keyword.SimpleDocumentStore() and no persist( anywhere in the file.doc.id_ = str(uuid4()) / loader without filename_as_id.datetime.now() / time.time() written into document metadata before
ingestion.docstore_strategy=DocstoreStrategy.DUPLICATES_ONLY on a live-sync pipeline.The same "hash → docstore → insert/update/skip" contract across the common
stacks. Compact below; full runnable code in references/R2-cross-framework.md.
Verified against current docs (May 2026).
IngestionPipeline(docstore=..., vector_store=..., docstore_strategy=DocstoreStrategy.UPSERTS_AND_DELETE). Hash is on the
document (content + doc_id); pipeline.run(documents=docs) returns only the
nodes it processed — empty list = clean no-op. Persist the docstore
(Redis/Mongo/Postgres in prod) to keep idempotency across restarts.index(docs, SQLRecordManager(...), vectorstore, cleanup="full", source_id_key="source"). The RecordManager is the
docstore-equivalent (a hash per source_id in a durable store). Returns
{num_added, num_updated, num_skipped, num_deleted}; on an unchanged re-run
num_skipped == len(docs) and the rest are 0. cleanup="full" propagates
deletes for a complete snapshot, "incremental" for source-scoped batches.{doc_id: sha256(text)} table; per doc:
prev == h → SKIP, prev set → delete-then-add (UPDATE), else INSERT; after
the loop purge stored_keys − seen (DELETE); persist(). This is exactly
what the two frameworks automate.The bottom row is the anti-pattern (A1). Every correct row has the same shape: a persisted hash ledger separate from the vector store, consulted before embedding, with an explicit delete-propagation switch.
DocstoreStrategy):
https://developers.llamaindex.ai/python/framework/module_guides/loading/ingestion_pipeline/cleanup):
https://python.langchain.com/docs/how_to/indexing/references/R1-source-evidence.md — claims traced to base [[llamaindex]] skill + primary docs.references/R2-cross-framework.md — full LlamaIndex / LangChain / manual code with delete-propagation variants.intermediate/operation_candidates.json — OP-01..09 in machine-readable Trigger / Action / Output / Evidence form.[[llamaindex]] — base RAG SOP. This overlay enhances OP-08
(IngestionWithDocstore) and anti-pattern A10 with the full
re-ingest-correctness contract.|
---name: agentsop-idempotent-ingestiondescription: |---# skills/agentsop-idempotent-ingestion/SKILL.mdSSS
skills/agentsop-idempotent-ingestion/SKILL.md dosyasını yayıncıdan indirin veya yolu kopyalayıp depoyu açın. PlugMCP beceriyi listeler, dosyayı barındırmaz.
Yayıncı bunu şurada tutuyor: https://github.com/agentsope/SkillAlchemy.
Claude, Cursor, Codex ve Grok Build, yayıncının deposundan eklediğiniz bir SKILL.md dosyasını okuyabilir. Grok Bot, becerinin GitHub klasör bağlantısını bir sohbette gönderdiğinizde onu kurar.
| Old vectors were embedded by the old model; mixed space breaks retrieval |
| Full re-embed; tag index with model name+version ([[llamaindex]] A2) |