APPLYDJINN
← ETL and Data Engineering: reliable pipelinesProgress & review ↗

Typorithm · medium

Bounded retries and stable logical batch identity

pythonetldata-engineeringindependent-practice
Saved on this device
Full lesson · 4 learning steps Open any exercise for focused practice
1

Step 1 of 4 · intro card

Bounded retries and stable logical batch identity

0%

Concept and contract

Retry transient failures within a finite budget and keep logical batch identity independent of attempts.

Retries address failures that may succeed when attempted again, but retrying every error can amplify a permanent problem. This fixture distinguishes success, transient failure and permanent failure, requires a positive attempt budget and records a finite exponential delay plan. It does not wait in real time or contact a live service.

The logical batch key is a SHA-256 digest of the same raw UTF-8 payload for every attempt. An attempt number identifies execution history, while the batch key identifies intended work. Keeping them separate supports deduplication at a cooperating sink; generating a new logical identity on each retry would defeat that purpose.

Mechanism and worked example

A scripted adapter supplies outcomes from the fixture.

A scripted adapter supplies outcomes from the fixture. Each attempted operation records the stable key, then success stops with a committed status, permanent failure stops immediately, and transient failure schedules another attempt only if budget remains. Planned delays grow as powers of two between attempts.

The returned report includes attempt count, final status, key consistency, raw byte count and delays. Follow a permanent failure followed by a scripted success: the success must never be consumed. Another case exhausts the budget on transient failures, and a third succeeds after two transient outcomes with no delay added after completion.

Complete worked example

python

from hashlib import sha256

def inspect_example(outcomes, maximum, raw):
    if type(maximum) is not int or maximum <= 0 or not outcomes:
        raise ValueError('positive budget and scripted outcomes required')
    batch_key = sha256(raw.encode('utf-8')).hexdigest()
    keys, delays, seen = [], [], []
    status = 'failed'
    for attempt in range(maximum):
        kind = outcomes[min(attempt, len(outcomes)-1)]
        keys.append(batch_key)
        seen.append(kind)
        if kind == 'success':
            status = 'committed'
            break
        if kind != 'transient':
            break
        if attempt + 1 < maximum:
            delays.append(2 ** attempt)
    return {'attempts':len(keys),'status':status,'sameBatchKey':len(set(keys)) == 1,
            'sourceBytes':len(raw.encode('utf-8')),'delays':delays}
Reasoning
A scripted adapter supplies outcomes from the fixture. Each attempted operation records the stable key, then success stops with a committed status, permanent failure stops immediately, and transient failure schedules another attempt only if budget remains. Planned delays grow as powers of two between attempts.
Download the ETL and Data Engineering computer lab ↗

Focused decision: use the complete example for context

python

if kind != 'transient':
    break
if attempt + 1 < maximum:
    delays.append(2 ** attempt)
Decision
Sum the planned retry delays while preserving one logical batch identity and stopping on permanent failures.
Download the ETL and Data Engineering computer lab ↗

Counterexamples and common mistakes

The mutation changes which failure category stops the loop.

The mutation changes which failure category stops the loop. A permanent problem is retried and can reach a later scripted success, violating the declared policy. A happy-path first-attempt success would not reveal that defect, so classification and exhaustion cases belong in the contract.

A stable key does not force an arbitrary external service to deduplicate. The receiver must implement and retain an appropriate idempotency contract. Real retry policies may also need jitter, server hints, deadlines and cancellation. This lesson records deterministic planned delays to verify control flow; it does not claim those values are universally appropriate for production traffic.

Controlled bug: observe the rejected data contract

python

from hashlib import sha256

def inspect_example(outcomes, maximum, raw):
    if type(maximum) is not int or maximum <= 0 or not outcomes:
        raise ValueError('positive budget and scripted outcomes required')
    batch_key = sha256(raw.encode('utf-8')).hexdigest()
    keys, delays, seen = [], [], []
    status = 'failed'
    for attempt in range(maximum):
        kind = outcomes[min(attempt, len(outcomes)-1)]
        keys.append(batch_key)
        seen.append(kind)
        if kind == 'success':
            status = 'committed'
            break
        if kind == 'unknown':
            break
        if attempt + 1 < maximum:
            delays.append(2 ** attempt)
    return {'attempts':len(keys),'status':status,'sameBatchKey':len(set(keys)) == 1,
            'sourceBytes':len(raw.encode('utf-8')),'delays':delays}
Reasoning
The mutation changes which failure category stops the loop. A permanent problem is retried and can reach a later scripted success, violating the declared policy. A happy-path first-attempt success would not reveal that defect, so classification and exhaustion cases belong in the contract.
Download the ETL and Data Engineering computer lab ↗

Independent practice

Add delayBudget, the sum of the recorded planned delays.

Add delayBudget, the sum of the recorded planned delays. Preserve attempt count, failure classification and stable batch identity. The value represents the deterministic schedule requested by this fixture, not observed wall-clock latency, network time or an assurance that a real request completed within a deadline.

Calculate the sum after the retry loop rather than adding a delay on every iteration. A permanent first failure has no retry delay, and an exhausted budget has no extra sleep after the last allowed attempt. Run all declared cases so the total cannot accidentally include a delay after success or after the budget has already ended.

Step 1: Sum the planned retry delays while preserving one logical batch identity and stopping on permanent failures. Step 2: Edit the owned exercise file and run verify.py --practice etl-retry. Preserve caller inputs and the independently declared failure cases.

Practice starter: solve before reading the solution

python

from hashlib import sha256

def inspect_example(outcomes, maximum, raw):
    if type(maximum) is not int or maximum <= 0 or not outcomes:
        raise ValueError('positive budget and scripted outcomes required')
    batch_key = sha256(raw.encode('utf-8')).hexdigest()
    keys, delays, seen = [], [], []
    status = 'failed'
    for attempt in range(maximum):
        kind = outcomes[min(attempt, len(outcomes)-1)]
        keys.append(batch_key)
        seen.append(kind)
        if kind == 'success':
            status = 'committed'
            break
        if kind != 'transient':
            break
        if attempt + 1 < maximum:
            delays.append(2 ** attempt)
    return {'attempts':len(keys),'status':status,'sameBatchKey':len(set(keys)) == 1,
            'sourceBytes':len(raw.encode('utf-8')),'delays':delays}

# Independent task: Sum the planned retry delays while preserving one logical batch identity and stopping on permanent failures.
Reasoning
Sum the planned retry delays while preserving one logical batch identity and stopping on permanent failures.
Download the ETL and Data Engineering computer lab ↗
  • Hint 1: Add delayBudget, the sum of the recorded planned delays. Preserve attempt count, failure classification and stable batch identity. The value represents the deterministic schedule requested by this fixture, not observed wall-clock latency, network time or an assurance that a real request completed within a deadline.
  • Hint 2: Calculate the sum after the retry loop rather than adding a delay on every iteration. A permanent first failure has no retry delay, and an exhausted budget has no extra sleep after the last allowed attempt. Run all declared cases so the total cannot accidentally include a delay after success or after the budget has already ended.

Solution, tests and trade-offs

The solution reports the aggregate delay plan while keeping retries bounded.

The solution reports the aggregate delay plan while keeping retries bounded. Exact observations compare delays as well as their sum, because two different schedules can share a total. The fixture's raw byte count also keeps content identity tied to the actual encoded payload instead of a mutable attempt label.

When integrating a real adapter, translate concrete error responses into reviewed transient and permanent categories. Authentication failure, malformed input and throttling need different treatment. Cap total elapsed time as well as attempt count, and preserve enough telemetry to correlate attempts with one logical batch without logging private payloads or credentials.

Practice solution and acceptance checks

python

from hashlib import sha256

def inspect_example(outcomes, maximum, raw):
    if type(maximum) is not int or maximum <= 0 or not outcomes:
        raise ValueError('positive budget and scripted outcomes required')
    batch_key = sha256(raw.encode('utf-8')).hexdigest()
    keys, delays, seen = [], [], []
    status = 'failed'
    for attempt in range(maximum):
        kind = outcomes[min(attempt, len(outcomes)-1)]
        keys.append(batch_key)
        seen.append(kind)
        if kind == 'success':
            status = 'committed'
            break
        if kind != 'transient':
            break
        if attempt + 1 < maximum:
            delays.append(2 ** attempt)
    return {'attempts':len(keys),'status':status,'sameBatchKey':len(set(keys)) == 1,
            'sourceBytes':len(raw.encode('utf-8')),'delays':delays, 'delayBudget':sum(delays)}
Reasoning
The solution reports the aggregate delay plan while keeping retries bounded. Exact observations compare delays as well as their sum, because two different schedules can share a total. The fixture's raw byte count also keeps content identity tied to the actual encoded payload instead of a mutable attempt label.
Download the ETL and Data Engineering computer lab ↗

Interview reasoning and next step

Explain why retry identity and execution attempt identity should differ.

Explain why retry identity and execution attempt identity should differ. Then trace the permanent, exhausted and eventual-success paths, identifying exactly where another attempt is allowed. Separate a planned backoff schedule from measured service latency and from the sink's actual idempotency behavior.

For a stretch, inject a deterministic jitter source and a deadline into the owned adapter, then verify cancellation before the next attempt. Keep waits simulated in tests. A robust retry policy should be inspectable through state transitions rather than relying on long sleeping tests or accidental success from an unreliable external endpoint.

Optional stretch — after the core task works: Inject deterministic jitter and a deadline while keeping the owned verification free of real waits.

Computer practice: https://applydjinn.com/typorithm/etl-data-engineering-lab-20261107.zip. Use an isolated Python 3.13 environment, install requirements.txt and run verify.py. Edit exercises/etl-retry.py and run --practice etl-retry. Each case owns its files and database; Parquet uses the pinned DuckDB engine. The recovery probe terminates a separate interpreter and reopens the SQLite file. The app retains static examples, prepared results and questions without an editor or code execution. Use the course Download button once to save every lesson and illustration; opening an unsaved course does not automatically download it.

Retry transient failures within a finite budget and keep logical batch identity independent of attempts.