APPLYDJINN
← ETL and Data Engineering: reliable pipelinesProgress & review ↗

Typorithm · medium

Quality gates and quarantine budgets

pythonetldata-engineeringindependent-practice
Saved on this device
Full lesson · 4 learning steps Open any exercise for focused practice
1

Step 1 of 4 · intro card

Quality gates and quarantine budgets

0%

Concept and contract

Combine independent publication requirements, reconcile rejected rows and define an empty-dataset policy.

A quality gate decides whether a candidate dataset is safe to publish under declared rules. The fixture validates nonempty identities and exact nonnegative integer cents, separates invalid rows, checks accepted identity uniqueness, enforces a maximum bad-row percentage and applies an explicit empty-input policy.

These conditions are conjunctive: each required property must hold. A low invalid fraction cannot excuse duplicate accepted identities, and uniqueness cannot excuse an exceeded quarantine budget. The threshold uses integer arithmetic over counts, avoiding a floating-point boundary decision when an exact percentage can be compared directly.

Mechanism and worked example

The program first divides rows into accepted and rejected collections.

The program first divides rows into accepted and rejected collections. It then calculates uniqueness, bad-row budget and nonempty policy independently, returning both observations and canPublish. The budget compares rejected count times one hundred with total count times the configured percentage.

One case exceeds the budget, another has empty input with publication disallowed, and another duplicates an otherwise valid identity. The boundary case has exactly the allowed percentage and must pass that particular gate. This separation makes the reason for a failed publication visible instead of burying every rule inside one opaque boolean expression.

Complete worked example

python

def inspect_example(rows, maximum_bad_percent, allow_empty):
    if type(maximum_bad_percent) is not int or not 0 <= maximum_bad_percent <= 100:
        raise ValueError('integer percentage required')
    accepted = [r for r in rows if isinstance(r.get('id'), str) and r['id'] and type(r.get('amount_cents')) is int and r['amount_cents'] >= 0]
    bad = len(rows) - len(accepted)
    unique = len({r['id'] for r in accepted}) == len(accepted)
    within_budget = bad * 100 <= len(rows) * maximum_bad_percent
    nonempty = bool(rows) or allow_empty
    can_publish = unique and within_budget and nonempty
    return {'accepted': len(accepted), 'quarantined': bad, 'canPublish': can_publish}
Reasoning
The program first divides rows into accepted and rejected collections. It then calculates uniqueness, bad-row budget and nonempty policy independently, returning both observations and canPublish. The budget compares rejected count times one hundred with total count times the configured percentage.
Download the ETL and Data Engineering computer lab ↗

Focused decision: use the complete example for context

python

can_publish = unique and within_budget and nonempty
Decision
Report an exact accepted-value aggregate while requiring every independent publication gate to pass.
Download the ETL and Data Engineering computer lab ↗

Counterexamples and common mistakes

The mutation joins conditions using or instead of and.

The mutation joins conditions using or instead of and. A dataset can now publish merely because one convenient property passes, even while another required gate fails. A valid-looking total or a nonempty result is not a substitute for the complete publication contract.

Filtering bad rows before calculating the denominator can also disguise source quality. The budget in this lesson is based on the original delivery count, not only the accepted subset. A production quarantine needs source lineage, reasons, retention and an operational owner; tolerating a bounded fraction does not mean those records can disappear without reconciliation.

Controlled bug: observe the rejected data contract

python

def inspect_example(rows, maximum_bad_percent, allow_empty):
    if type(maximum_bad_percent) is not int or not 0 <= maximum_bad_percent <= 100:
        raise ValueError('integer percentage required')
    accepted = [r for r in rows if isinstance(r.get('id'), str) and r['id'] and type(r.get('amount_cents')) is int and r['amount_cents'] >= 0]
    bad = len(rows) - len(accepted)
    unique = len({r['id'] for r in accepted}) == len(accepted)
    within_budget = bad * 100 <= len(rows) * maximum_bad_percent
    nonempty = bool(rows) or allow_empty
    can_publish = unique or within_budget or nonempty
    return {'accepted': len(accepted), 'quarantined': bad, 'canPublish': can_publish}
Reasoning
The mutation joins conditions using or instead of and. A dataset can now publish merely because one convenient property passes, even while another required gate fails. A valid-looking total or a nonempty result is not a substitute for the complete publication contract.
Download the ETL and Data Engineering computer lab ↗

Independent practice

Add acceptedCents, the sum of amounts in accepted canonical rows.

Add acceptedCents, the sum of amounts in accepted canonical rows. Preserve the gate decisions and counts, including the exact budget boundary and configured empty policy. The aggregate should be reported even when publication is blocked, so an operator can inspect candidate data without mistaking a report for authorization to publish.

Calculate the sum from accepted rows after validation. Do not include rejected negative or boolean amounts, and do not silently deduplicate accepted identities as a side effect. Duplicate detection is a gate in this contract. The independent cases distinguish reporting candidate totals from changing the dataset until it happens to pass.

Step 1: Report an exact accepted-value aggregate while requiring every independent publication gate to pass. Step 2: Edit the owned exercise file and run verify.py --practice etl-quality. Preserve caller inputs and the independently declared failure cases.

Practice starter: solve before reading the solution

python

def inspect_example(rows, maximum_bad_percent, allow_empty):
    if type(maximum_bad_percent) is not int or not 0 <= maximum_bad_percent <= 100:
        raise ValueError('integer percentage required')
    accepted = [r for r in rows if isinstance(r.get('id'), str) and r['id'] and type(r.get('amount_cents')) is int and r['amount_cents'] >= 0]
    bad = len(rows) - len(accepted)
    unique = len({r['id'] for r in accepted}) == len(accepted)
    within_budget = bad * 100 <= len(rows) * maximum_bad_percent
    nonempty = bool(rows) or allow_empty
    can_publish = unique and within_budget and nonempty
    return {'accepted': len(accepted), 'quarantined': bad, 'canPublish': can_publish}

# Independent task: Report an exact accepted-value aggregate while requiring every independent publication gate to pass.
Reasoning
Report an exact accepted-value aggregate while requiring every independent publication gate to pass.
Download the ETL and Data Engineering computer lab ↗
  • Hint 1: Add acceptedCents, the sum of amounts in accepted canonical rows. Preserve the gate decisions and counts, including the exact budget boundary and configured empty policy. The aggregate should be reported even when publication is blocked, so an operator can inspect candidate data without mistaking a report for authorization to publish.
  • Hint 2: Calculate the sum from accepted rows after validation. Do not include rejected negative or boolean amounts, and do not silently deduplicate accepted identities as a side effect. Duplicate detection is a gate in this contract. The independent cases distinguish reporting candidate totals from changing the dataset until it happens to pass.

Solution, tests and trade-offs

The solution adds an observation while leaving the decision rules intact.

The solution adds an observation while leaving the decision rules intact. Empty accepted input sums to zero, and a failed publication remains failed even if acceptedCents looks reasonable. Exact assertions cover the individual gates and combined result, exposing the controlled disjunction defect.

Quality requirements should fit the dataset. A reference table may require zero rejected rows; an event feed may permit a small bounded quarantine with alerts. Define thresholds before inspecting a bad delivery, version the rules, and record enough evidence to explain why one generation was published while another remained staged for investigation.

Practice solution and acceptance checks

python

def inspect_example(rows, maximum_bad_percent, allow_empty):
    if type(maximum_bad_percent) is not int or not 0 <= maximum_bad_percent <= 100:
        raise ValueError('integer percentage required')
    accepted = [r for r in rows if isinstance(r.get('id'), str) and r['id'] and type(r.get('amount_cents')) is int and r['amount_cents'] >= 0]
    bad = len(rows) - len(accepted)
    unique = len({r['id'] for r in accepted}) == len(accepted)
    within_budget = bad * 100 <= len(rows) * maximum_bad_percent
    nonempty = bool(rows) or allow_empty
    can_publish = unique and within_budget and nonempty
    return {'accepted': len(accepted), 'quarantined': bad, 'canPublish': can_publish, 'acceptedCents': sum(r['amount_cents'] for r in accepted)}
Reasoning
The solution adds an observation while leaving the decision rules intact. Empty accepted input sums to zero, and a failed publication remains failed even if acceptedCents looks reasonable. Exact assertions cover the individual gates and combined result, exposing the controlled disjunction defect.
Download the ETL and Data Engineering computer lab ↗

Interview reasoning and next step

Explain why all required gates must pass and how the empty-input policy affects a full replacement.

Explain why all required gates must pass and how the empty-input policy affects a full replacement. Publishing an accidental empty snapshot can remove previously visible data even when no individual row is invalid. Counts, uniqueness and completeness therefore answer different questions about the same candidate generation.

For a stretch, return named failure reasons and reconcile accepted plus rejected counts with the source total. Add a required partition-presence check in a separate exercise. Keep evaluation deterministic and avoid allowing a successful aggregate to override a failed identity or completeness requirement.

Optional stretch — after the core task works: Return named failure reasons and add a separately specified required-partition completeness gate.

Computer practice: https://applydjinn.com/typorithm/etl-data-engineering-lab-20261107.zip. Use an isolated Python 3.13 environment, install requirements.txt and run verify.py. Edit exercises/etl-quality.py and run --practice etl-quality. Each case owns its files and database; Parquet uses the pinned DuckDB engine. The recovery probe terminates a separate interpreter and reopens the SQLite file. The app retains static examples, prepared results and questions without an editor or code execution. Use the course Download button once to save every lesson and illustration; opening an unsaved course does not automatically download it.

Combine independent publication requirements, reconcile rejected rows and define an empty-dataset policy.