Pattern · last reviewed 2026-09-25

Dedup is not acknowledgement

Context & problem

A pipeline that watches an upstream source for change — a cron that diffs a feed, a queue consumer, a webhook receiver, a nightly job that re-scores a corpus — carries two jobs that read as one. The first is deciding whether this input is new work: a duplicate check, an idempotency key, an hasOpenDraft-style guard. The second is remembering that you looked at this input, so the same unchanged state doesn't re-trigger forever: a cursor, a Kafka offset, an SQS message deletion, a snapshot hash, a "last seen" timestamp. Call the second one the watermark, whatever shape it takes in your system.

Keeping those two jobs separate is easy to state and easy to get wrong in code, because the natural place to write "we've seen this" is right after the dedup check runs — whether or not the check actually let the item through. When the watermark write doesn't ask "did this get consumed?", a duplicate check that correctly rejects a second write also, as a side effect, tells the watermark to move on. The input is gone: not processed, not queued, not retried. Just marked as handled.

Forces

• At-least-once delivery is a deliberate design, and it only holds if the commit ordering is right. Kafka's consumer docs describe exactly this shape for a database-writing consumer: "we will manually commit the offsets only after the corresponding records have been inserted into the database" — because committing earlier means "records would be considered consumed after they were returned to the user in poll()," so a crash between poll and insert loses them. The same logic applies whether the boundary is a Kafka offset, an SQS message, or a hand-rolled snapshot column.
• "Nothing happened" and "we dropped it" produce the same log line. A cron that reports outcome: ok every day looks identical whether it is genuinely idle or quietly discarding flagged changes — unless something counts the discards separately from the no-ops.
• The loss doesn't announce itself; it waits. A source that changes rarely may not flag again for weeks, so the gap between "the item was silently dropped" and "someone notices the output never updated" can be long enough that nobody connects the two.
• A dedup guard and a watermark write are usually authored at different times, by different reasoning. The guard is added to stop duplicate rows; the watermark write is added (often earlier) to stop reprocessing. Neither change, read on its own, looks wrong — the bug is in the combination, which is exactly why review rarely catches it.

The pattern

Split "is this a duplicate" from "did we durably take this" into two checks with two different failure modes, and let only the second one move the watermark:

• Check for duplicates before doing the expensive work (a model call, a fetch, a transform), not only before the final write — a duplicate caught late still cost you the work, even if it never costs you a second row.
• Advance the watermark only for items a downstream consumer durably took — enqueued, inserted, committed, deleted from the queue. An item held back by the duplicate check is left exactly where it was, so the same change re-surfaces on the next cycle instead of vanishing.
• Make "nothing happened" observable. Count suppressed/held items separately from processed items, and log or surface the count. A run that processed 0 and suppressed 0 is idle. A run that processed 0 and suppressed 4 is blocked — those are different operational states and a single "ran OK" boolean erases the difference.

This is the same ordering both vendors' docs recommend — process, then acknowledge — applied one level up, to whatever cursor or snapshot your own pipeline uses to avoid reprocessing. (Kafka's own default is auto-commit on a timer, not this; the manual, commit-after-process pattern is what its docs recommend for correctness, not what a consumer gets for free.)

A minimal sketch

The bug, generically — the watermark write doesn't ask whether the item made it past the dedup check:

// WRONG — dedup guards the write, not the watermark
for (const item of diffs) {
  if (await isDuplicate(item.key)) { suppressed++; continue; }
  await consume(item);            // enqueue, insert, process
}
for (const item of diffs) {
  await advanceWatermark(item);   // runs for EVERY diff, including suppressed ones
}

The fix keeps the same two loops, but the second one skips whatever the first one held back:

// RIGHT — a suppressed item holds the watermark back too
const held = new Set();
for (const item of diffs) {
  if (await isDuplicate(item.key)) { held.add(item.key); suppressed++; continue; }
  await consume(item);
}
for (const item of diffs) {
  if (held.has(item.key)) continue;  // re-checked next run, no re-spend
  await advanceWatermark(item);
}
return { processed: diffs.length - held.size, suppressed };

The cost of the fix is one cheap read per held item per cycle (the duplicate check itself, run again) instead of one write. That is the trade you want: a repeated no-op check is inert, a lost input is not.

Trade-offs

• A permanently-open duplicate blocks its lane forever. If nothing ever resolves the in-flight item the duplicate check is waiting on, the source behind it never re-snapshots and never retries — this pattern stops silent loss, it does not stop indefinite starvation. Pair it with staleness alerting on whatever "in-flight" state can go stale.
• A suppressed-count nobody reads is the same silence with more code. Emitting the counter is not the control; a dashboard, a log-based alert, or a huddle report that actually looks at it is.
• This is not a retry-with-backoff pattern. It is about not discarding input state on a suppressed cycle — it doesn't by itself make a failing downstream call safe to retry, and mixing the two concerns is how a "just retry it" fix quietly reintroduces the original bug.
• Re-checking the duplicate guard every cycle has a cost too — for a very hot key with a very cheap check this is free; for an expensive existence check at high volume, the steady-state read cost of a stuck lane is worth measuring, not assuming.

When not to use this

Skip the split when the downstream action is genuinely idempotent and cheap to repeat — re-marking a boolean flag, re-writing a value that is already correct — because there advancing the watermark eagerly costs nothing extra even on a duplicate. Skip it too when the source re-emits its full state on every poll rather than an incremental diff: with no cursor to protect, there is nothing for a duplicate check to silently erase. And skip it when the suppressed item has an explicit, accepted expiry — a stale price quote, a cache entry past its TTL — where dropping it and advancing anyway is the correct behaviour, not a bug wearing the same shape as one.

How to test for it

A test that only asserts "no duplicate row was written" will pass on the broken version — the sink looks correct. Assert the source state instead: seed an in-flight duplicate, run the pipeline once, and check that the watermark/cursor/snapshot for the blocked item did not move. Then resolve the duplicate, run again, and assert the same item now gets consumed and the watermark advances. If your suite only ever exercises the free-lane path, add the blocked-lane case explicitly — it is the one the dedup check exists for, and the one most likely to be missing.

As-built evidence

aiArch runs a daily cron (runScheduled, src/routes/editorial.ts:166-262) that diffs a fixed list of upstream sources — vendor changelogs, pricing pages — against a stored snapshot, drafts a proposed content edit for anything changed, and enqueues it to an editorial review queue for a human decision at /admin. hasOpenDraft (src/lib/db.ts:168-180) exists to stop the same page/source pair from accumulating a second undecided draft while the first is still open — a correct, deliberate dedup guard.

Before the fix (Q-643/Q-649, diagnosed 2026-09-14), the cron's snapshot step re-stamped every changed source as seen once processing finished, regardless of whether hasOpenDraft had just rejected its draft. The result: a change behind an already-open draft still cost a model call to draft, was then discarded at insert time, and was recorded as seen by the snapshot write in the next step, which never consulted the dedup result. Once that happened, the change could not re-flag — the stored snapshot already matched it. The cron's own reporting never showed an error: the schedule fired on time every day, `outcome: ok`, and the only visible symptom was silence, which reads exactly like a quiet upstream. It took a direct database read cross-referencing open drafts in review_queue against `corpus_snapshots.fetched_at` to find that changes from two watched sources (AWS What's New, the Cloudflare changelog) had been drafted, discarded, and swallowed — see docs/ops/2026-09-14-watcher-silence-diagnosis.md for the full diagnosis, including the false leads ruled out along the way: a suspected mis-detecting regex in the change-narrowing logic, an initial "the watcher isn't running" verdict that a direct database read overturned, and a D1 read error that turned out to be specific to one session, not a real access block on the account.

The fix (docs/2026-09-14-q649-watcher-fix.md) moves the duplicate check earlier — before the draft is even generated, not only before the insert — and adds a blockedUrls set that step 6's snapshot loop explicitly skips (src/routes/editorial.ts:180-194, step 3; :231-252, step 6). A blocked source is left un-snapshotted on purpose, so the exact same change re-flags on the next run once the open draft is decided. ScheduledResult.skippedDuplicate is the suppressed-count this pattern calls for — it's returned from every run alongside `enqueued` and logged on every invocation (src/index.ts:849). But per Q-650 (open), that log line has never once been seen in Workers Observability across a retained week of otherwise-healthy cron activity, so today the count is produced, not read: our own instance of the trade-off above ("a suppressed-count nobody reads is the same silence with more code"). test/editorial-scheduled.test.ts pins six cases including the one this bug lived in: a blocked lane gets no drafter call and no snapshot, and re-flags on the next run once its open draft is resolved. The fix shipped as Worker 29ceb524; changes already swallowed before it deployed stayed lost until the same upstream source changed again — the pattern prevents the next occurrence, it does not recover what a stale snapshot had already erased.

Changelog

• 2026-09-25 — Initial publication, drawn from our own Q-643/Q-649 incident. Kafka manual-offset-commit semantics verified against the live KafkaConsumer javadoc; AWS SQS at-least-once/visibility-timeout semantics verified against the live SQS developer guide.