A counter that counted
past the finish line

Why a scraping run could report 602 successes out of 600 tasks — and the one-line-of-thinking fix that stops it.

→ / space / click to advance  ·  ← to go back

Production runs occasionally showed successful > total. Physically impossible — a run has 600 tasks, it can't succeed 602 times.

The number wasn't wrong data. It was the same task counted twice.

One task → one +1 on the run row

A bare, commutative ADD on DynamoDB. Fast, lock-free — and assumes each task settles exactly once.

A message can be delivered twice

If a scrape takes longer than the visibility timeout (slow target site), SQS assumes A died and hands the message to B. Now two workers hold the same task.

Both workers settle. Both add +1.

The check and the write weren't atomic

// HandleMessage — the "guard" that was supposed to dedupe
switch task.Status {
case Pending:                 // first delivery  → proceed
case Processing:              // redelivery      → proceed (!)
case Successful, Failed:      // already done    → drop ✓
    return nil
}
                    ▲ read here …
        … gateway call (slow) …
                    ▼ write terminal + ADD here — nothing held in between

The terminal drop only catches a sequential redelivery (A fully done before B reads). Two concurrent deliveries both read "not terminal" and sail through.

Make the write itself
the gatekeeper.

Instead of "read status, then blindly overwrite", the terminal write becomes a conditional write: "set me to successful — but only if I'm still processing."

Evaluated server-side inside the single DynamoDB PutItem. No version field, no extra read.

Only the CAS winner counts

Both still make one gateway call — that's inherent to at-least-once. Only the counting is now exactly-once.

Claim and settle are both conditional

Why a status CAS — not the other tools

• ✗
  Version field on the task row. Optimistic lock ⇒ a read-then-write on every settle. The spec explicitly rejected this cost; task rows stay unversioned (result keys are deterministic anyway).
• ✗
  SQS FIFO / content-dedup queue. Dedup window is 5 min; a slow scrape outlives it. Also loses per-run queue throughput.
• ✗
  Separate idempotency table. Another row + TTL to maintain, for a fact the task row already knows: its own status.
• ✓
  Conditional PutItem on status. Zero extra reads, zero new rows — the write we already do simply refuses to run twice.

What else the CAS quietly fixes

• ↻
  Retry path. A duplicate that hits a transient failure won't re-enqueue a second attempt or double-count spend — it loses the same CAS.
• ⟲
  Terminal-row revert. B can no longer clobber A's successful row back to processing and lose the result key.
• =
  Sequential redelivery. Still handled by the original terminal-status drop — the CAS is purely the concurrent-window backstop.
• ∑
  Run completion. Already safe — it used a versioned CAS. Only the counters were unguarded.

Real worker · DynamoDB Local · 5× concurrent dup

total       = 1
completed   = 5
successful  = 5
→ FAIL: 5 > 1

total       = 1
completed   = 1
successful  = 1
→ PASS ✓

go test -tags=integration -race -run DoubleCount  ·  success + failure paths  ·  full suite green

The bug wasn't the ADD.
It was who was allowed to run it.

github.com/ZenRows/conveyor · PR #166 · branch fix/cor-stats-double-count