← ETL and Data Engineering: reliable pipelines Progress & review ↗

Typorithm · medium

Immutable raw input and lineage

pythonetldata-engineeringindependent-practice

Saved on this device

Full lesson · 4 learning steps Open any exercise for focused practice

Step 1 of 4 · intro card

Immutable raw input and lineage

Concept and contract

Keep original source bytes, derive a content identity and separate replayable raw data from transformations.

A pipeline should be able to explain which input produced a table. The bronze fixture keeps the exact UTF-8 bytes supplied to it, including whitespace and line endings. Its SHA-256 digest supplies a content identity rather than a business record identity.

Two deliveries with identical bytes share one owned path. Different serialization can create different objects even when parsed JSON would describe the same record. That distinction is useful for audits: raw storage preserves what arrived, while later layers decide what it means.

Mechanism and worked example

The example encodes text once, calculates its digest, and writes the payload beneath an owned bronze directory.

The example encodes text once, calculates its digest, and writes the payload beneath an owned bronze directory. A second reference to the same payload resolves to the same path. The report independently reads the stored bytes and compares them with the original input.

Follow the leading space and trailing newline in the first acceptance case. They are not decorative after ingestion: both contribute to the digest and byte count. The Unicode case also distinguishes byte length from character count, while empty input remains a valid empty object.

Complete worked example

python

from hashlib import sha256
from pathlib import Path

def inspect_example(text):
    payload = text.encode('utf-8')
    revision = sha256(payload).hexdigest()
    path = Path('bronze') / (revision + '.jsonl')
    path.parent.mkdir(exist_ok=True)
    if not path.exists():
        path.write_bytes(payload)
    repeated = Path('bronze') / (sha256(payload).hexdigest() + '.jsonl')
    return {'sameObject': path == repeated, 'bytes': path.stat().st_size,
            'roundTrip': path.read_bytes().decode('utf-8') == text,
            'objects': len(list(path.parent.glob('*.jsonl')))}

Reasoning: The example encodes text once, calculates its digest, and writes the payload beneath an owned bronze directory. A second reference to the same payload resolves to the same path. The report independently reads the stored bytes and compares them with the original input.

Download the ETL and Data Engineering computer lab ↗

Focused decision: use the complete example for context

python

revision = sha256(payload).hexdigest()
path = Path('bronze') / (revision + '.jsonl')

Decision: Verify stored bytes against their content-addressed object name without normalizing the raw source.

Download the ETL and Data Engineering computer lab ↗

Counterexamples and common mistakes

The controlled bug strips the input before encoding it.

The controlled bug strips the input before encoding it. This looks like harmless cleanup but silently changes the evidence being stored. The leading space and newline disappear, so the byte count and recovered content no longer satisfy the raw contract.

Content addressing alone does not implement concurrent creation, access control, retention or authenticity. The fixture is a single-writer teaching directory, and its existence check is not a production lock. A hash detects content differences; it does not prove who supplied the data or whether the business record is correct.

Controlled bug: observe the rejected data contract

python

from hashlib import sha256
from pathlib import Path

def inspect_example(text):
    payload = text.strip().encode('utf-8')
    revision = sha256(payload).hexdigest()
    path = Path('bronze') / (revision + '.jsonl')
    path.parent.mkdir(exist_ok=True)
    if not path.exists():
        path.write_bytes(payload)
    repeated = Path('bronze') / (sha256(payload).hexdigest() + '.jsonl')
    return {'sameObject': path == repeated, 'bytes': path.stat().st_size,
            'roundTrip': path.read_bytes().decode('utf-8') == text,
            'objects': len(list(path.parent.glob('*.jsonl')))}

Reasoning: The controlled bug strips the input before encoding it. This looks like harmless cleanup but silently changes the evidence being stored. The leading space and newline disappear, so the byte count and recovered content no longer satisfy the raw contract.

Download the ETL and Data Engineering computer lab ↗

Independent practice

Extend the report with verified, a boolean calculated by hashing the bytes actually read from disk and comparing that digest with the intended revision.

Extend the report with verified, a boolean calculated by hashing the bytes actually read from disk and comparing that digest with the intended revision. Preserve the existing identity, byte count and round-trip fields. The added observation should come from stored content rather than a constant.

Find the readback payload before changing any write logic. Use the same hashing algorithm but a separate calculation over that payload. Run the normal, empty and Unicode cases so that verification cannot accidentally rely on a particular ASCII string or nonempty file.

Step 1: Verify stored bytes against their content-addressed object name without normalizing the raw source. Step 2: Edit the owned exercise file and run verify.py --practice etl-raw. Preserve caller inputs and the independently declared failure cases.

Practice starter: solve before reading the solution

python

from hashlib import sha256
from pathlib import Path

def inspect_example(text):
    payload = text.encode('utf-8')
    revision = sha256(payload).hexdigest()
    path = Path('bronze') / (revision + '.jsonl')
    path.parent.mkdir(exist_ok=True)
    if not path.exists():
        path.write_bytes(payload)
    repeated = Path('bronze') / (sha256(payload).hexdigest() + '.jsonl')
    return {'sameObject': path == repeated, 'bytes': path.stat().st_size,
            'roundTrip': path.read_bytes().decode('utf-8') == text,
            'objects': len(list(path.parent.glob('*.jsonl')))}

# Independent task: Verify stored bytes against their content-addressed object name without normalizing the raw source.

Reasoning: Verify stored bytes against their content-addressed object name without normalizing the raw source.

Download the ETL and Data Engineering computer lab ↗

Hint 1: Extend the report with verified, a boolean calculated by hashing the bytes actually read from disk and comparing that digest with the intended revision. Preserve the existing identity, byte count and round-trip fields. The added observation should come from stored content rather than a constant.
Hint 2: Find the readback payload before changing any write logic. Use the same hashing algorithm but a separate calculation over that payload. Run the normal, empty and Unicode cases so that verification cannot accidentally rely on a particular ASCII string or nonempty file.

Solution, tests and trade-offs

The solution retains byte-for-byte ingestion and adds an independent digest comparison at read time.

The solution retains byte-for-byte ingestion and adds an independent digest comparison at read time. This is a useful distinction between naming an object and validating its contents. The tests also preserve the caller's text and reject unexpected output from the example.

A real ingestion manifest would additionally record source name, retrieval time, schema hint and authorization context. Those fields need their own retention and privacy policies. Do not embed a changing retrieval timestamp into the content digest if your intended identity is the stable source payload itself.

Practice solution and acceptance checks

python

from hashlib import sha256
from pathlib import Path

def inspect_example(text):
    payload = text.encode('utf-8')
    revision = sha256(payload).hexdigest()
    path = Path('bronze') / (revision + '.jsonl')
    path.parent.mkdir(exist_ok=True)
    if not path.exists():
        path.write_bytes(payload)
    repeated = Path('bronze') / (sha256(payload).hexdigest() + '.jsonl')
    return {'sameObject': path == repeated, 'bytes': path.stat().st_size,
            'roundTrip': path.read_bytes().decode('utf-8') == text,
            'objects': len(list(path.parent.glob('*.jsonl'))), 'verified': sha256(path.read_bytes()).hexdigest() == revision}

Reasoning: The solution retains byte-for-byte ingestion and adds an independent digest comparison at read time. This is a useful distinction between naming an object and validating its contents. The tests also preserve the caller's text and reject unexpected output from the example.

Download the ETL and Data Engineering computer lab ↗

Interview reasoning and next step

Explain why equal business records can have unequal raw hashes, then identify which stage should normalize them.

Explain why equal business records can have unequal raw hashes, then identify which stage should normalize them. Raw ingestion should preserve evidence; canonical record comparison belongs to a transformation with an explicit schema and normalization policy. Be precise about which identity a downstream checkpoint records.

For a larger system, add a manifest mapping one logical delivery to several immutable objects and validate every referenced digest before transformation. Keep the experiment inside owned fixtures. The app displays these complete programs and prepared results; replaying files happens only in the downloaded computer lab.

Optional stretch — after the core task works: Add an owned manifest with source and content hash, then reject a deliberately corrupted object.

Computer practice: https://applydjinn.com/typorithm/etl-data-engineering-lab-20261107.zip. Use an isolated Python 3.13 environment, install requirements.txt and run verify.py. Edit exercises/etl-raw.py and run --practice etl-raw. Each case owns its files and database; Parquet uses the pinned DuckDB engine. The recovery probe terminates a separate interpreter and reopens the SQLite file. The app retains static examples, prepared results and questions without an editor or code execution. Use the course Download button once to save every lesson and illustration; opening an unsaved course does not automatically download it.

Keep original source bytes, derive a content identity and separate replayable raw data from transformations.