Deterministic deduplication and conflicting versions

Duplicate deliveries are normal in retryable ingestion. Choosing one row requires an ordering contract stronger than whichever record happens to arrive last. This fixture uses a source-controlled version per identity, retaining the greatest version regardless of input order. Output is sorted by identity for deterministic inspection.

Equal versions with different amounts are conflicts, not harmless duplicates. The pipeline raises ValueError instead of silently choosing a payload according to arrival timing. Version comparison is meaningful only under the declared source agreement; a random ingestion timestamp cannot replace a trustworthy per-record version without changing the semantics.

For each row, the program looks up the current retained record. A missing identity or strictly newer version replaces that record with a copied dictionary. A matching version is checked for incompatible payload. Older versions leave the retained record unchanged, and the report returns identities in sorted order.

Follow a newer update that appears before an older delivery of the same identity. The older record cannot roll the value back. Copying rows keeps the result from reusing caller-owned dictionaries, while the checker independently verifies the original arguments stay unchanged. Empty input yields a valid empty deduplicated result.

The mutation reverses the version comparison. It can retain an older value after a newer one has already been observed, making the result depend on a stale delivery. A repeated identical payload may still pass, which is why the acceptance suite uses different versions and amounts.

Arbitrary tie-breaking can be even harder to detect. Sorting by ingestion order or taking the last dictionary value makes an ambiguous source appear valid. Preserve conflict evidence and decide whether the entire batch fails or the identity is quarantined. Do not silently reinterpret equal-version conflicts as updates without an explicit source rule.

Extend the report with discarded, the number of source rows not represented by the final one-row-per-identity output. Preserve highest-version selection, deterministic ordering and rejection of conflicting equal versions. The task counts redundant or superseded deliveries; it does not add those rows back into the canonical result.

Compute the difference between input count and retained identity count after successful validation. Keep conflict handling before returning a success report. Compare empty input, several versions for one identity and multiple distinct identities. A constant discard count or a count based only on immediately adjacent duplicates will fail these independent cases.

Step 1: Keep the highest source version and report how many duplicate or obsolete records were discarded. Step 2: Edit the owned exercise file and run verify.py --practice etl-dedupe. Preserve caller inputs and the independently declared failure cases.

The solution adds reconciliation metadata without changing the selected business state. The number discarded includes older versions and repeated identical deliveries, so its name must not be mistaken for a count of invalid records. Conflicts still interrupt the operation rather than becoming an apparently successful discard.

A database sink needs the same ordering rule across separate batches; in-memory deduplication alone protects only the current delivery. The later UPSERT lesson persists the version comparison. If versions reset or identities are reused upstream, introduce a generation or namespace into the source contract before relying on numeric ordering across those boundaries.

Explain how deduplication differs from aggregation and why a version is part of the data contract. An update stream should preserve current state; summing duplicate amounts would implement a different operation. Identify what evidence lets an operator distinguish harmless redelivery from an equal-version source contradiction.

For a stretch, shuffle a valid delivery repeatedly and assert identical output, then introduce a same-version conflict that every ordering must reject. Keep the random seed fixed if you automate this experiment. Determinism should cover both successful results and explicit rejection, not merely the presentation order of rows.

Optional stretch — after the core task works: Permute delivery order and verify stable selection plus order-independent conflict rejection.

Computer practice: https://applydjinn.com/typorithm/etl-data-engineering-lab-20261107.zip. Use an isolated Python 3.13 environment, install requirements.txt and run verify.py. Edit exercises/etl-dedupe.py and run --practice etl-dedupe. Each case owns its files and database; Parquet uses the pinned DuckDB engine. The recovery probe terminates a separate interpreter and reopens the SQLite file. The app retains static examples, prepared results and questions without an editor or code execution. Use the course Download button once to save every lesson and illustration; opening an unsaved course does not automatically download it.