← ETL and Data Engineering: reliable pipelines Progress & review ↗

Typorithm · medium

Bounded batches and the final partial chunk

pythonetldata-engineeringindependent-practice

Saved on this device

Full lesson · 4 learning steps Open any exercise for focused practice

Step 1 of 4 · intro card

Bounded batches and the final partial chunk

Concept and contract

Consume an iterator in bounded groups without dropping its last records.

Batching limits how many input records a transformation handles together. It does not automatically bound the whole program's memory, because an accumulated output list or the original source collection may still grow. The fixture isolates consumption into groups of a positive exact integer size.

A final group smaller than the requested size is still data. It should normally be processed rather than discarded. Empty input produces no groups, and zero or negative sizes are invalid configuration. These boundaries make batching predictable before it is connected to database transactions, network requests or a streaming source.

Mechanism and worked example

The program obtains an iterator and repeatedly takes at most size records with islice.

The program obtains an iterator and repeatedly takes at most size records with islice. A nonempty slice becomes the next batch; an empty slice ends the loop. This preserves source order and retains the remaining records even when they do not fill a complete group.

With five records and size two, the groups have lengths two, two and one. Trace exactly when the iterator advances: a chunk is consumed once and no record should appear in two groups. The worked return value keeps batches for inspection; a real sink could write each group and release it instead.

Complete worked example

python

from itertools import islice

def inspect_example(values, size):
    if type(size) is not int or size <= 0:
        raise ValueError('positive batch size required')
    consumed = []
    def source():
        for value in values:
            consumed.append(value)
            yield value
    stream = source()
    batches = []
    while batch := list(islice(stream, size)):
        batches.append(batch)
    return {'batches': batches, 'consumed': consumed}

Reasoning: The program obtains an iterator and repeatedly takes at most size records with islice. A nonempty slice becomes the next batch; an empty slice ends the loop. This preserves source order and retains the remaining records even when they do not fill a complete group.

Download the ETL and Data Engineering computer lab ↗

Focused decision: use the complete example for context

python

while batch := list(islice(stream, size)):
    write_batch(batch)

Decision: Retain the final short batch and add an explicit count to each emitted batch report.

Download the ETL and Data Engineering computer lab ↗

Counterexamples and common mistakes

The controlled bug adds a full-size condition before keeping a batch.

The controlled bug adds a full-size condition before keeping a batch. The last one-record group is consumed from the iterator but omitted from output. No exception signals this data loss, and a source whose length is divisible by the size would hide the defect.

A separate mistake would materialize the entire iterator before batching, defeating the intended bound for large sources. The fixture cannot prove global memory limits from small test inputs; it demonstrates a local consumption rule. State whether downstream buffering, retries and retained results can expand the effective memory budget.

Controlled bug: observe the rejected data contract

python

from itertools import islice

def inspect_example(values, size):
    if type(size) is not int or size <= 0:
        raise ValueError('positive batch size required')
    consumed = []
    def source():
        for value in values:
            consumed.append(value)
            yield value
    stream = source()
    batches = []
    while batch := list(islice(stream, size)):
        if len(batch) < size: break
        batches.append(batch)
    return {'batches': batches, 'consumed': consumed}

Reasoning: The controlled bug adds a full-size condition before keeping a batch. The last one-record group is consumed from the iterator but omitted from output. No exception signals this data loss, and a source whose length is divisible by the size would hide the defect.

Download the ETL and Data Engineering computer lab ↗

Independent practice

Change the report so each emitted group is a dictionary with rows and count.

Change the report so each emitted group is a dictionary with rows and count. The count must be calculated from that actual group, including the final partial one. Preserve order, empty-input behavior and invalid-size rejection. The baseline list-only output should fail the new independent acceptance contract.

Keep the islice loop and modify the emitted representation. Calculate len(batch), not the configured size, because the final group can be shorter. Use the empty and nondivisible cases before adding more data. Avoid mutating the caller's list merely to make the groups look independent; consumption and ownership are different concerns.

Step 1: Retain the final short batch and add an explicit count to each emitted batch report. Step 2: Edit the owned exercise file and run verify.py --practice etl-batches. Preserve caller inputs and the independently declared failure cases.

Practice starter: solve before reading the solution

python

from itertools import islice

def inspect_example(values, size):
    if type(size) is not int or size <= 0:
        raise ValueError('positive batch size required')
    consumed = []
    def source():
        for value in values:
            consumed.append(value)
            yield value
    stream = source()
    batches = []
    while batch := list(islice(stream, size)):
        batches.append(batch)
    return {'batches': batches, 'consumed': consumed}

# Independent task: Retain the final short batch and add an explicit count to each emitted batch report.

Reasoning: Retain the final short batch and add an explicit count to each emitted batch report.

Download the ETL and Data Engineering computer lab ↗

Hint 1: Change the report so each emitted group is a dictionary with rows and count. The count must be calculated from that actual group, including the final partial one. Preserve order, empty-input behavior and invalid-size rejection. The baseline list-only output should fail the new independent acceptance contract.
Hint 2: Keep the islice loop and modify the emitted representation. Calculate len(batch), not the configured size, because the final group can be shorter. Use the empty and nondivisible cases before adding more data. Avoid mutating the caller's list merely to make the groups look independent; consumption and ownership are different concerns.

Solution, tests and trade-offs

The solution wraps every batch with a measured row count.

The solution wraps every batch with a measured row count. For the last group, the count is one rather than the requested size two. Exact nested type checks distinguish the new dictionaries from the previous plain lists and verify the count alongside the retained values.

When batching real writes, decide what constitutes a retry unit and when a source position can advance. A group that was consumed is not necessarily committed. Database transaction boundaries, request byte limits and per-row validation may require smaller units than the configured record count, so retain enough lineage to retry without skipping data.

Practice solution and acceptance checks

python

from itertools import islice

def inspect_example(values, size):
    if type(size) is not int or size <= 0:
        raise ValueError('positive batch size required')
    consumed = []
    def source():
        for value in values:
            consumed.append(value)
            yield value
    stream = source()
    batches = []
    while batch := list(islice(stream, size)):
        batches.append({'rows': batch, 'count': len(batch)})
    return {'batches': batches, 'consumed': consumed}

Reasoning: The solution wraps every batch with a measured row count. For the last group, the count is one rather than the requested size two. Exact nested type checks distinguish the new dictionaries from the previous plain lists and verify the count alongside the retained values.

Download the ETL and Data Engineering computer lab ↗

Interview reasoning and next step

Explain why a bounded chunk size is useful but insufficient evidence of a bounded pipeline.

Explain why a bounded chunk size is useful but insufficient evidence of a bounded pipeline. Include the source, transformation, sink buffer and retained diagnostics in your memory argument. Then describe how a partial group would be handled if processing stops at the end of a finite delivery.

For a stretch, replace the retained output with a callback sink that records only group lengths in an owned test. Add a generator that counts consumption to verify no extra record is read. Keep the final partial case: it is often the simplest way to expose a silent truncation bug.

Optional stretch — after the core task works: Use an owned sink callback and consumption counter to verify local batching without retaining every row.

Computer practice: https://applydjinn.com/typorithm/etl-data-engineering-lab-20261107.zip. Use an isolated Python 3.13 environment, install requirements.txt and run verify.py. Edit exercises/etl-batches.py and run --practice etl-batches. Each case owns its files and database; Parquet uses the pinned DuckDB engine. The recovery probe terminates a separate interpreter and reopens the SQLite file. The app retains static examples, prepared results and questions without an editor or code execution. Use the course Download button once to save every lesson and illustration; opening an unsaved course does not automatically download it.

Consume an iterator in bounded groups without dropping its last records.