← ETL and Data Engineering: reliable pipelines Progress & review ↗

Typorithm · medium

CSV records, quoting and structural errors

pythonetldata-engineeringindependent-practice

Saved on this device

Full lesson · 4 learning steps Open any exercise for focused practice

Step 1 of 4 · intro card

CSV records, quoting and structural errors

Concept and contract

Parse logical records with the CSV library instead of splitting physical lines or commas.

CSV is a record format with quoting rules, not simply a list of strings separated at every comma. A quoted description can contain commas and even newlines. Header names and field counts are another contract, separate from the conversion of amount_cents into an integer.

The fixture uses an in-memory text stream with newline handling delegated to the CSV reader. It accepts exactly the declared three-column header and rejects malformed record shapes. This keeps a shifted column from being quietly interpreted as an amount or an identity.

Mechanism and worked example

DictReader associates each parsed field with its header after resolving quoting.

DictReader associates each parsed field with its header after resolving quoting. The worked program checks the header, validates missing or extra fields, converts cents, and returns canonical row lists. Output order follows the source records rather than an unrelated alphabetical sort.

Trace the quoted description in the first case: its internal comma remains part of one field. The second case demonstrates a signed amount, and the empty-body case yields no rows after its valid header. A header-only input is structurally valid even though a later quality gate may disallow publishing an empty dataset.

Complete worked example

python

import csv
import io

def inspect_example(text):
    reader = csv.DictReader(io.StringIO(text, newline=''))
    if reader.fieldnames != ['id', 'description', 'amount_cents']:
        raise ValueError('unexpected header')
    rows = []
    for row in reader:
        if None in row or any(value is None for value in row.values()):
            raise ValueError('wrong field count')
        rows.append([row['id'], row['description'], int(row['amount_cents'])])
    return rows

Reasoning: DictReader associates each parsed field with its header after resolving quoting. The worked program checks the header, validates missing or extra fields, converts cents, and returns canonical row lists. Output order follows the source records rather than an unrelated alphabetical sort.

Download the ETL and Data Engineering computer lab ↗

Focused decision: use the complete example for context

python

reader = csv.DictReader(io.StringIO(text, newline=''))

Decision: Parse quoted CSV records correctly, then filter negative amounts after extraction rather than editing raw delimiters.

Download the ETL and Data Engineering computer lab ↗

Counterexamples and common mistakes

The counterexample splits text into lines and then splits each line at commas.

The counterexample splits text into lines and then splits each line at commas. A quoted description is divided into separate pieces, so the acceptance case observes the wrong description. This can corrupt meaning even when the program still returns something shaped like a row.

A parser does not replace validation. Quoting support cannot guarantee unique identifiers, valid business ranges or a suitable schema version. Keep structural parsing errors visible and decide separately whether an invalid delivery fails as a whole or sends individual records into quarantine with their source positions retained.

Controlled bug: observe the rejected data contract

python

def inspect_example(text):
    rows = []
    for line in text.splitlines()[1:]:
        fields = line.split(',')
        rows.append([fields[0], fields[1], int(fields[-1])])
    return rows

Reasoning: The counterexample splits text into lines and then splits each line at commas. A quoted description is divided into separate pieces, so the acceptance case observes the wrong description. This can corrupt meaning even when the program still returns something shaped like a row.

Download the ETL and Data Engineering computer lab ↗

Independent practice

Change the exercise so it retains only records with nonnegative amount_cents after parsing and conversion.

Change the exercise so it retains only records with nonnegative amount_cents after parsing and conversion. Keep proper quoted-field handling, exact headers and the same canonical output row lists. The signed input in the second case now becomes a useful boundary test for the new filtering policy.

Locate the point where cents has already become an integer. Apply the predicate there rather than comparing source strings, because lexical ordering would not model numeric ordering. Preserve zero, avoid modifying the caller's text, and retain the baseline behavior for valid quoted descriptions and a header-only delivery.

Step 1: Parse quoted CSV records correctly, then filter negative amounts after extraction rather than editing raw delimiters. Step 2: Edit the owned exercise file and run verify.py --practice etl-csv. Preserve caller inputs and the independently declared failure cases.

Practice starter: solve before reading the solution

python

import csv
import io

def inspect_example(text):
    reader = csv.DictReader(io.StringIO(text, newline=''))
    if reader.fieldnames != ['id', 'description', 'amount_cents']:
        raise ValueError('unexpected header')
    rows = []
    for row in reader:
        if None in row or any(value is None for value in row.values()):
            raise ValueError('wrong field count')
        rows.append([row['id'], row['description'], int(row['amount_cents'])])
    return rows

# Independent task: Parse quoted CSV records correctly, then filter negative amounts after extraction rather than editing raw delimiters.

Reasoning: Parse quoted CSV records correctly, then filter negative amounts after extraction rather than editing raw delimiters.

Download the ETL and Data Engineering computer lab ↗

Hint 1: Change the exercise so it retains only records with nonnegative amount_cents after parsing and conversion. Keep proper quoted-field handling, exact headers and the same canonical output row lists. The signed input in the second case now becomes a useful boundary test for the new filtering policy.
Hint 2: Locate the point where cents has already become an integer. Apply the predicate there rather than comparing source strings, because lexical ordering would not model numeric ordering. Preserve zero, avoid modifying the caller's text, and retain the baseline behavior for valid quoted descriptions and a header-only delivery.

Solution, tests and trade-offs

The solution filters the canonical parsed rows using an explicit nonnegative predicate.

The solution filters the canonical parsed rows using an explicit nonnegative predicate. That is a domain policy layered after a structural parser, making the reason for each transformation understandable. Independent solution cases reject the untouched baseline because it still includes the negative record.

Silently dropping records is not always acceptable in production. A financial ingestion may require a rejection report instead. The small exercise isolates the filtering mechanism; a deployable pipeline should reconcile counts and record why each excluded row was removed, especially when downstream totals are used for business decisions.

Practice solution and acceptance checks

python

import csv
import io

def inspect_example(text):
    reader = csv.DictReader(io.StringIO(text, newline=''))
    if reader.fieldnames != ['id', 'description', 'amount_cents']:
        raise ValueError('unexpected header')
    rows = []
    for row in reader:
        if None in row or any(value is None for value in row.values()):
            raise ValueError('wrong field count')
        rows.append([row['id'], row['description'], int(row['amount_cents'])])
    return [row for row in rows if row[2] >= 0]

Reasoning: The solution filters the canonical parsed rows using an explicit nonnegative predicate. That is a domain policy layered after a structural parser, making the reason for each transformation understandable. Independent solution cases reject the untouched baseline because it still includes the negative record.

Download the ETL and Data Engineering computer lab ↗

Interview reasoning and next step

Explain why opening a real CSV file with appropriate newline handling matters and why splitting physical lines loses the distinction between a line and a quoted record.

Explain why opening a real CSV file with appropriate newline handling matters and why splitting physical lines loses the distinction between a line and a quoted record. Then describe which failures belong to parsing, type conversion and business validation. These stages need different remediation information.

For a stretch, add an owned fixture with a newline inside a quoted description and another with an extra column. Check the parsed description and explicit rejection separately. Do not treat a passing comma-containing example as proof that every malformed CSV dialect has the same interpretation.

Optional stretch — after the core task works: Add quoted-newline and extra-field fixtures, with separate accepted and rejected expectations.

Computer practice: https://applydjinn.com/typorithm/etl-data-engineering-lab-20261107.zip. Use an isolated Python 3.13 environment, install requirements.txt and run verify.py. Edit exercises/etl-csv.py and run --practice etl-csv. Each case owns its files and database; Parquet uses the pinned DuckDB engine. The recovery probe terminates a separate interpreter and reopens the SQLite file. The app retains static examples, prepared results and questions without an editor or code execution. Use the course Download button once to save every lesson and illustration; opening an unsaved course does not automatically download it.

Parse logical records with the CSV library instead of splitting physical lines or commas.